3D lane line marking method, electronic equipment and storage medium

Abstract

The invention provides a 3D lane line marking method, electronic equipment and a storage medium, and the method comprises the steps: obtaining a target image, and obtaining a current pose, the target image being an image including a lane line, and the current pose being a pose of a camera when the target image is collected; according to the current pose, extracting a 3D lane line marking point set matched with the current pose from a local 3D map associated with the target image; performing lane line prediction on the target image to obtain a predicted lane line point set; projecting the 3D lane line marking point set into the target image to obtain a projected lane line point set; and according to an error between the projection lane line point set and the prediction lane line point set, correcting the 3D lane line marking point set to obtain a corrected 3D lane line marking point set. According to the method, the error of the 3D map is corrected and eliminated, the problem of 3D map lane line projection is solved, the same 3D map can be reused, and the marking cost is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of lane line marking, and in particular to a 3D lane line marking method, electronic device and storage medium. Background Art

[0002] The current mainstream lane detection solution is based on deep learning models, and the lane detection accuracy of deep learning models depends on the lane marking accuracy. In addition to traditional road-level map content, 3D maps (i.e. high-precision maps) also contain accurate lane lines, road reference lines and other information, so 3D maps can be used as truth values ​​to train deep learning models.

[0003] However, 3D maps usually require a dedicated collection vehicle to collect data such as LiDAR and integrated navigation, and then perform data processing and road feature extraction. The production cost is high and the process is complicated. In addition, due to the errors in the 3D map itself and the trajectory positioning errors, there are still errors when projecting it onto the 2D image, which in turn affects the lane line inspection accuracy of the deep learning model. Therefore, how to achieve low-cost and high-precision 3D lane line annotation is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of the above problems, embodiments of the present application provide a 3D lane marking method, an electronic device, and a storage medium to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of an embodiment of the present application, a 3D lane marking method is disclosed, the method comprising:

[0006] Acquire a target image, and acquire a current posture, wherein the target image is an image including lane lines, and the current posture is the posture of the camera when acquiring the target image;

[0007] According to the current posture, extracting a 3D lane line annotation point set matching the current posture from a local 3D map associated with the target image;

[0008] Performing lane line prediction on the target image to obtain a predicted lane line point set;

[0009] Projecting the 3D lane line annotation point set into the target image to obtain a projected lane line point set;

[0010] According to the error between the projected lane line point set and the predicted lane line point set, the 3D lane line annotation point set is corrected to obtain a corrected 3D lane line annotation point set.

[0011] In a second aspect of an embodiment of the present application, a 3D lane marking device is disclosed, the device comprising:

[0012] An acquisition module, used to acquire a target image and a current posture, wherein the target image is an image including lane lines, and the current posture is the posture of the camera when acquiring the target image;

[0013] An extraction module, configured to extract, according to the current posture, a 3D lane line annotation point set matching the current posture from a local 3D map associated with the target image;

[0014] A prediction module, used to perform lane line prediction on the target image to obtain a predicted lane line point set;

[0015] A projection module, used to project the 3D lane line annotation point set into the target image to obtain a projected lane line point set;

[0016] The correction module is used to correct the 3D lane line annotation point set according to the error between the projected lane line point set and the predicted lane line point set to obtain a corrected 3D lane line annotation point set.

[0017] A third aspect of an embodiment of the present application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the 3D lane line marking method described in the first aspect of the embodiment of the present application are implemented.

[0018] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the steps of the 3D lane line marking method described in the first aspect of the embodiment of the present application are implemented.

[0019] According to a fifth aspect of an embodiment of the present application, a computer program product is disclosed, including a computer program, which, when executed by a processor, implements the steps of the 3D lane line marking method described in the first aspect of an embodiment of the present application.

[0020] The embodiments of the present application include the following advantages:

[0021] In an embodiment of the present application, a 3D lane line annotation point set matching the current posture is extracted from a local 3D map associated with a target image according to the current posture, and lane lines are predicted for the target image to obtain a predicted lane line point set, and then the 3D lane line annotation point set is projected into the target image to obtain a projected lane line point set, and finally, based on the error between the projected lane line point set and the predicted lane line point set, the 3D lane line annotation point set is corrected to obtain a corrected 3D lane line annotation point set.

[0022] Since the corrected 3D lane line annotation point set is obtained by correcting the prediction results of the target image based on the 3D map, the error of the 3D map is eliminated, and the lane line projection problem of the 3D map is effectively solved. At the same time, since the 3D map itself has a lane line annotation point set, there is no need for manual annotation. A high-precision 3D lane line annotation point set can be automatically obtained through correction. Moreover, the same 3D map can be corrected to obtain 3D lane line annotation point sets for multiple trajectories based on different target images and corresponding postures, so that the 3D map can be reused, thereby reducing the annotation cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 This is a flowchart of a 3D lane marking method provided by an embodiment of the present application;

[0025] Figure 2 It is a schematic diagram of a projected lane line point set and a predicted lane line point set of an uncorrected 3D lane line annotation point set provided in an embodiment of the present application;

[0026] Figure 3 It is a schematic diagram of a projected lane line point set and a predicted lane line point set of a modified 3D lane line annotation point set provided in an embodiment of the present application;

[0027] Figure 4 It is a flowchart of the steps of a 3D lane marking method for adding a reference point set provided in an embodiment of the present application;

[0028] Figure 5 is a flowchart of another 3D lane marking method provided in an embodiment of the present application;

[0029] Figure 6 It is a schematic diagram of the overall framework of a 3D lane marking method provided in an embodiment of the present application;

[0030] Figure 7 It is a structural schematic diagram of a 3D lane marking device provided in an embodiment of the present application;

[0031] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] Reference Figure 1 As shown, Figure 1 This is a flowchart of a 3D lane marking method provided by an embodiment of the present application. Figure 1 As shown, a 3D lane marking method provided in an embodiment of the present application may include steps S110 to S150:

[0034] Step S110: Acquire a target image, and acquire a current posture, wherein the target image is an image including lane lines, and the current posture is the posture of the camera when acquiring the target image.

[0035] Among them, the target image can be an image directly captured by the camera, or a video frame image extracted from the video captured by the camera. The pose refers to the position and posture of the camera. The current pose is collected by the vehicle's inertial navigation sensor. When collecting the target image, the camera pose (i.e., the current pose) at the corresponding moment of the target image is also collected. In the specific implementation, for a section of road with a 3D map, the camera and the vehicle's inertial navigation sensor are used to collect multiple lane line images (target images) and the poses at the corresponding moments, so that the 3D map can be corrected based on the multiple lane line images and the camera poses at the corresponding moments, and a large number of corrected 3D lane line annotation point sets can be obtained.

[0036] Step S120: According to the current posture, extracting a 3D lane line annotation point set matching the current posture from the local 3D map associated with the target image.

[0037] Among them, a 3D map refers to a high-precision map of a section of road, in which the lane lines and road reference information of the section are marked. The lane lines and road reference information in the 3D map are marked with reference to the world coordinate system, that is, the positions of the lane lines and road reference information are marked with longitude and latitude. A local 3D map is a 3D map within a preset range with the current posture as a reference. The local 3D map is determined in the following way: the 3D map is searched according to the current posture and the preset search range to obtain a local 3D map. The 3D lane line annotation point set is a 3D lane line annotation point set in a local coordinate system (i.e., the vehicle coordinate system) with a position point on the vehicle as the origin.

[0038] Specifically, extracting a 3D lane line annotation point set matching the current posture from a local 3D map associated with the target image includes: performing convergence angle compensation and coordinate transformation processing on the local 3D map to obtain a 3D lane line annotation point set matching the current posture, wherein convergence angle compensation refers to preprocessing such as repairing and verifying the lane line, reconstructing bifurcation points and segmentation points, so as to obtain a complete lane line, thereby ensuring the availability of the lane line; coordinate transformation refers to converting the 3D lane line annotation from the world coordinate system to the local coordinate system (i.e., the vehicle coordinate system).

[0039] Step S130: performing lane line prediction on the target image to obtain a predicted lane line point set.

[0040] Specifically, a pre-trained 2D lane line perception model is used to perform lane line prediction processing on the target image to obtain a lane line prediction result corresponding to the target image. The lane line prediction result is represented by a predicted lane line point set, which is a 2D lane line point set.

[0041] Step S140: Projecting the 3D lane line annotation point set into the target image to obtain a projected lane line point set.

[0042] Specifically, the dimension of the 3D lane line annotation point set is different from that of the predicted lane line point set. For the convenience of analysis, the 3D lane line annotation point set needs to be projected to the same dimension as the predicted lane line point set, that is, projected to the target image. Due to different projection methods corresponding to different postures, for example, the lane line image 1 and lane line image 2 collected by the camera at posture A and posture B are different, and then the method of projecting the 3D lane line annotation point set to lane line image 1 and lane line image 2 is also different. In order to obtain an accurate projected lane line point set, the projection method is determined according to the current posture, and then the 3D lane line annotation point set is projected into the target image according to the determined projection method to obtain the projected lane line point set, which is a 2D lane line point set. The projection method can be understood as the conversion matrix between the local coordinate system (i.e., the vehicle coordinate system) and the camera coordinate system of the camera that collects the target image (i.e., the vehicle-mounted camera installed on the vehicle). The different postures of the camera that collects the target image correspond to different camera coordinate systems and different projection methods.

[0043] Step S150: Correcting the 3D lane line annotation point set according to the error between the projected lane line point set and the predicted lane line point set to obtain a corrected 3D lane line annotation point set.

[0044] In the embodiment of the present application, due to the influence of the error of the 3D map itself and the trajectory positioning error (i.e., posture error), the projected lane line point set cannot completely coincide with the predicted lane line point set. Therefore, the 3D lane line annotation point set is corrected based on the error between the projected lane line point set and the predicted lane line point set to eliminate the error between the projected lane line point set and the predicted lane line point set, and a corrected 3D lane line annotation point set with higher accuracy is obtained. The projected lane line point set corresponding to the corrected 3D lane line annotation point set can coincide with the predicted lane line point set, and then training samples are generated based on the corrected 3D lane line annotation point set for training the 3D lane line prediction model.

[0045] For example, Figure 2 is a schematic diagram of a projected lane line point set and a predicted lane line point set of an uncorrected 3D lane line annotation point set provided in an embodiment of the present application, Figure 3 3D lane line annotation point set and predicted lane line point set. It can be seen that the projected lane line point set and predicted lane line point set corresponding to the corrected 3D lane line annotation point set can overlap.

[0046] In the embodiment of the present application, since the corrected 3D lane line annotation point set is obtained by correction based on the 3D map using the prediction results of the lane line image (target image), the error of the 3D map is eliminated, and the 3D map lane line projection problem is effectively solved. At the same time, since the 3D map itself has a lane line annotation point set, there is no need for manual annotation, and a high-precision 3D lane line annotation point set can be automatically obtained through correction. Moreover, the same 3D map can be corrected to obtain 3D lane line annotation point sets for multiple trajectories based on different target images and corresponding postures, thereby realizing the reuse of 3D maps and reducing annotation costs.

[0047] In an optional embodiment, after obtaining the corrected 3D lane line annotation point set, the method further includes steps S160 to S180:

[0048] Step S160: Based on the predicted lane line point set, the corrected 3D lane line annotation point set is evaluated to obtain a first evaluation result.

[0049] Step S170: performing 3D line type evaluation on the corrected 3D lane line annotation point set to obtain a second evaluation result.

[0050] Step S180: Based on the first evaluation result and / or the second evaluation result, the corrected 3D lane line annotation point set is cleaned according to a target threshold to obtain a cleaned 3D lane line annotation point set.

[0051] Among them, evaluating the corrected 3D lane line annotation point set means: evaluating the error between the projection result of the corrected 3D lane line annotation point set and the predicted lane line point set. Evaluating the 3D line type of the corrected 3D lane line annotation point set means: evaluating the geometric shape of the 3D lane line. Cleaning the corrected 3D lane line annotation point set means: removing the corrected 3D lane line annotation point set with poor quality. Among them, the target threshold can be set according to different scenarios. For example, the vehicle has higher requirements for the accuracy of lane lines at close distances and relatively large requirements for the accuracy of lane lines at long distances, so different target thresholds are set for lane lines within different distance ranges of the vehicle. Finally, training samples are generated based on the cleaned 3D lane line annotation point set for training the 3D lane line prediction model.

[0052] In the embodiment of the present application, by evaluating and cleaning the corrected 3D lane line annotation point set, the 3D lane line annotation point set with poor quality is removed, thereby further ensuring the quality of the 3D lane line annotation point set used for model training.

[0053] In an optional embodiment, a reference point set is added to correct the 3D lane line annotation point set. Figure 4 As shown, Figure 4 4 is a flowchart of a method for adding a reference point set to a 3D lane line marking method provided in an embodiment of the present application, specifically including steps S410 to S480:

[0054] Step S410: Acquire a target image, and acquire a current posture, wherein the target image is an image including lane lines, and the current posture is the posture of the camera when the target image is acquired.

[0055] Step S420: Acquire a reference point set, wherein the 3D position of each reference point in the reference point set is predetermined, and each reference point in the reference point set is a pixel point used to characterize a reference object in the target image.

[0056] Step S430: projecting the reference point set into the target image to obtain a projected reference point set.

[0057] Step S440: Determine, from the target image, a set of pixel points representing each reference point.

[0058] Step S450: According to the current posture, extract a 3D lane line annotation point set matching the current posture from the local 3D map associated with the target image.

[0059] Step S460: Projecting the 3D lane line annotation point set into the target image to obtain a projected lane line point set.

[0060] Step S470: performing lane line prediction on the target image to obtain a predicted lane line point set.

[0061] Step S480: Correcting the 3D lane line annotation point set according to the error between the projected lane line point set and the predicted lane line point set, and the error between the projected reference point and the pixel point set.

[0062] In an embodiment of the present application, the reference point is other visual features in the environment (i.e., the pixel points of the reference object in the target image). For example, the reference object may be a building, and the reference point may be the pixel points of the building in the environment. The reference point set is projected as a projected reference point set in the same projection manner as the 3D lane line annotation point set. If there is an error between the projected reference point and the pixel point, this also reflects that there is an error in the 3D lane line annotation point set. Therefore, when correcting the error between the projected lane line point set and the predicted lane line point set, the error between the projected reference point and the pixel point set is also used to correct the 3D lane line annotation point set, thereby obtaining a more accurate corrected 3D lane line annotation point set.

[0063] Furthermore, the 3D position of any reference point in the reference point set is obtained according to the following steps: determining a set of pixel points used to represent the same reference point from the target image and the first N lane line images of the target image; and determining the 3D position of the reference point based on the set of pixel points and the posture of the camera when acquiring the target image and the first N lane line images.

[0064] In the embodiment of the present application, the pixel points representing the same reference point refer to the pixel points of the same object in the target image and the other lane line image, that is, the pixel points representing the same reference point are actually pixel pairs. For example, pixel A in the target image and pixel B in another lane line image, if pixel A and pixel B represent the same object (that is, the same reference point), then pixel A and pixel B constitute a pixel pair, and then the 3D position of the reference point is calculated based on the pixel points representing the same reference point, the corresponding posture of the target image and the posture of the corresponding lane line image, that is, the 3D position of the reference point is obtained by triangulation.

[0065] In an optional embodiment, after obtaining the projected lane line point set, it is also necessary to associate the projected lane line with the predicted lane line, and then modify the 3D lane line annotation point set based on the associated projected lane line and predicted lane line. Figure 5 As shown, Figure 5 This is a flowchart of another 3D lane marking method provided by an embodiment of the present application, which specifically includes steps S510 to S570:

[0066] Step S510: Acquire a target image, and acquire a current posture, wherein the target image is an image including lane lines, and the current posture is the posture of the camera when the target image is acquired.

[0067] Step S520: According to the current posture, extract a 3D lane line annotation point set matching the current posture from the local 3D map associated with the target image.

[0068] Step S530: performing lane line prediction on the target image to obtain a predicted lane line point set.

[0069] Step S540: Projecting the 3D lane line annotation point set into the target image to obtain a projected lane line point set.

[0070] Step S550: For each projected lane line formed by the projected lane line point set, calculate the distance between a point on the projected lane line and each predicted lane line formed by the predicted lane line point set.

[0071] Specifically, since both the projected lane line and the predicted lane line are represented by a point set, the distance between each projected lane line point and each predicted lane line can be calculated by the km algorithm.

[0072] Step S560: Determine the associated projected lane line and predicted lane line according to the calculated distance.

[0073] Specifically, the closest projected lane line and predicted lane line are determined as associated lane lines. For example, the distances between projected lane line 1 and predicted lane line 1, predicted lane line 2, predicted lane line 3, and predicted lane line 4 are 0.3, 2, 2.5, and 3, respectively, so projected lane line 1 and predicted lane line 1 are determined as associated lane lines.

[0074] Step S570: According to the error between the projected lane line point set of the associated projected lane line and the predicted lane line point set of the predicted lane line, the 3D lane line annotation point set is corrected to obtain a corrected 3D lane line annotation point set.

[0075] In the embodiment of the present application, the actual road section contains multiple lane lines, and accordingly, the projected lane line point set contains multiple projected lane lines, and the predicted lane line point set contains multiple predicted lane lines. Therefore, it is necessary to associate the projected lane line and the predicted lane line representing the same lane line, so that when calculating the error between the projected lane line point set and the predicted lane line point set of the predicted lane line, the error between the associated projected lane line and the predicted lane line is calculated.

[0076] In an optional embodiment, the 3D lane line annotation point set is corrected according to the error between the projected lane line point set and the predicted lane line point set, including:

[0077] Correcting the 3D lane line annotation point set at least according to a lane line reprojection error between the projected lane line point set and the predicted lane line point set;

[0078] The lane line reprojection error is used to characterize the error between every two matching point pairs in the projected lane line point set and the predicted lane line point set.

[0079] In an embodiment of the present application, when the 3D lane line annotation point set is corrected using the lane line reprojection error, the annotation position of the 3D lane line annotation point set is adjusted to reduce the error between the projected lane line point set and the predicted lane line point set. When the lane line reprojection error is zero or less than a preset error threshold, it indicates that the projected lane line point set coincides with the predicted lane line point set.

[0080] Furthermore, the lane line reprojection error is calculated in the following manner: for each projected lane line point in the projected lane line point set, a predicted lane line point closest to the projected lane line point is determined from the predicted lane line point set; and the lane line reprojection error is determined based on the distance between each projected lane line point in the projected lane line point set and the nearest predicted lane line point.

[0081] For example, for point A in the projected lane line point set, find point B from the predicted lane line point set that is closest to point A, and use the straight-line distance from point A to point B as the lane line reprojection error between point A and point B. Similarly, the remaining lane line points are calculated using the same method.

[0082] In an optional embodiment, the 3D lane line annotation point set is corrected according to the error between the projected lane line point set and the predicted lane line point set, including:

[0083] Correcting the 3D lane line annotation point set according to at least one of a lane line segmentation point error and a lane line bifurcation point error between the projected lane line point set and the predicted lane line point set, and the lane line reprojection error;

[0084] The lane line segmentation point error is used to characterize the error between the projected lane line segmentation point in the projected lane line point set and the predicted lane line segmentation point in the predicted lane line point set;

[0085] The lane line bifurcation point error is used to characterize the error between the projected lane line bifurcation point in the projected lane line point set and the predicted lane line bifurcation point in the predicted lane line point set.

[0086] In the embodiment of the present application, multiple errors are taken into consideration to correct the 3D lane line annotation point set, which not only realizes the correction of the 3D lane line reprojection error, but also supports the correction of road information such as segmentation points and bifurcation points. This method is easier to expand under different scenarios and line tasks, further eliminating the errors of 3D maps and solving the 3D map lane line projection problem. In the specific implementation, the following three situations are included: 1) Using the lane line segmentation point error and the lane line reprojection error to correct the 3D lane line annotation point set; 2) Using the lane line bifurcation point error and the lane line reprojection error to correct the 3D lane line annotation point set; 3) Using the lane line segmentation point error, the lane line bifurcation point error and the lane line reprojection error to correct the 3D lane line annotation point set.

[0087] The lane segmentation point refers to the position point where the lane line changes, for example, the position point where the lane line changes from a solid line to a dotted line is the lane segmentation point. When the lane segmentation point error is used to correct the 3D lane line annotation point set, the annotation position of the segmentation point in the 3D lane line annotation point set is adjusted to reduce the error between the projected lane line segmentation point and the predicted lane line segmentation point. When the lane segmentation point error is zero or less than the preset error threshold, it means that the projected lane line segmentation point coincides with the predicted lane line segmentation point.

[0088] Furthermore, the lane line segmentation point error is calculated in the following manner: for each projected lane line segmentation point in the projected lane line point set, a matching predicted lane line segmentation point is determined from the predicted lane line point set, and the matching projected lane line segmentation point and the predicted lane line segmentation point are used to characterize the same segmentation point on the lane line; and the lane line segmentation point error is determined based on the distance between each matching projected lane line segmentation point and the predicted lane line segmentation point.

[0089] For example, for the projected lane line segmentation point C, find the matching predicted lane line segmentation point D from the predicted lane line point set, and use the straight-line distance from segmentation point C to segmentation point D as the lane line segmentation point error between segmentation point C and segmentation point D. Similarly, the remaining segmentation points are calculated using the same method.

[0090] The lane line bifurcation point refers to the position point where the lane line bifurcates. For example, when the road bifurcates, the position point where one lane line is divided into two lane lines is the lane line bifurcation point. When the lane line bifurcation point error is used to correct the 3D lane line annotation point set, the annotation position of the bifurcation point in the 3D lane line annotation point set is adjusted to reduce the error between the projected lane line bifurcation point and the predicted lane line bifurcation point. When the lane line bifurcation point error is zero or less than the preset error threshold, it means that the projected lane line bifurcation point coincides with the predicted lane line bifurcation point.

[0091] Furthermore, the lane line segmentation point error is calculated in the following manner: for each projected lane line bifurcation point in the projected lane line point set, a matching predicted lane line bifurcation point is determined from the predicted lane line point set, and the matching projected lane line bifurcation point and the predicted lane line bifurcation point are used to characterize the same bifurcation point on the lane line; and the lane line bifurcation point error between each matching projected lane line bifurcation point and the predicted lane line bifurcation point is determined.

[0092] For a lane line with a bifurcation point, one lane line is divided into two lane lines starting from the bifurcation point, so the predicted lane line point set includes a first predicted lane line point set and a second predicted lane line point set, and the first predicted lane line point set and the second predicted lane line point set are used to characterize the two lane lines with the bifurcation point. In specific implementation, the lane line bifurcation point error is determined in a manner including but not limited to a first manner and a second manner. Specifically, determining the lane line bifurcation point error between each matched projected lane line bifurcation point and the predicted lane line bifurcation point according to the first manner means: determining the first predicted lane line bifurcation point closest to the projected lane line bifurcation point from the first predicted lane line point set, and determining the second predicted lane line bifurcation point closest to the projected lane line bifurcation point from the second predicted lane line point set; determining the first lane line bifurcation point error according to the first distance between the projected lane line bifurcation point and the nearest first predicted lane line bifurcation point, and the second distance between the projected lane line bifurcation point and the nearest second predicted lane line bifurcation point. For example, for the bifurcation point E of the projected lane line, the bifurcation point F closest to the bifurcation point E is determined from the first predicted lane line point set, and the bifurcation point G closest to the bifurcation point E is determined from the second predicted lane line point set, and then the bifurcation point error of the first lane line is obtained based on the distance between the bifurcation point E and the bifurcation point F, and the distance between the bifurcation point E and the bifurcation point G.

[0093] Determining the lane line bifurcation point error between each matched projected lane line bifurcation point and the predicted lane line bifurcation point according to the second method means: determining the same predicted lane line point in the first predicted lane line point set and the second predicted lane line point set as the predicted lane line bifurcation point; determining the second lane line bifurcation point error according to the distance between the projected lane line bifurcation point and the predicted lane line bifurcation point. The same predicted lane line point refers to the bifurcation point of two predicted lane lines. For example, for the bifurcation point H of the predicted lane line and the bifurcation point I of the projected lane line, the distance between the bifurcation point H and the bifurcation point I is used as the second lane line bifurcation point error.

[0094] In an optional embodiment, the 3D lane line annotation point set is corrected multiple times; according to the error between the predicted lane line point set and the projected lane line point set, the 3D lane line annotation point set is corrected to obtain a corrected 3D lane line annotation point set, including steps A1 to A4:

[0095] Step A1: After correcting the 3D lane line annotation point set once, a corrected 3D lane line annotation point set is obtained.

[0096] Step A2: Projecting each corrected 3D lane line annotation point set into the target image to obtain the corrected projected lane line point set.

[0097] Step A3: When the error between the projected lane line point set after this correction and the predicted lane line point set does not meet the target condition, proceed to the next correction.

[0098] The target condition refers to an error correction threshold, and the error not meeting the target condition refers to the error being greater than the error correction threshold.

[0099] Step A4: When the error between the projected lane line point set after this correction and the predicted lane line point set meets the target condition, the correction is stopped to obtain the 3D lane line annotation point set after the last correction.

[0100] The error meeting the target condition means that the error is not greater than the error correction threshold, at which point the correction can be ended.

[0101] In the embodiment of the present application, the correction of the 3D lane line annotation point set is an iterative optimization process. Each time the optimization variables of the 3D lane line annotation point set (for example, the annotation position of the 3D lane line annotation point set, the segmentation point annotation position and the bifurcation point annotation position) are corrected, the projection result in the target image will also change accordingly. Therefore, each time a correction is completed, a new projection is required, and a new error is calculated based on the projected lane line point set after this correction, so as to determine whether to make correction again or end the correction based on the new error.

[0102] In a specific implementation, the error between the predicted lane line point set and the projected lane line point set includes a lane line bifurcation point error, and the lane line bifurcation point error is used to characterize the error between the projected lane line bifurcation point in the projected lane line point set and the predicted lane line bifurcation point in the predicted lane line point set; the method of calculating the lane line bifurcation point error in two adjacent correction processes is different.

[0103] In the embodiment of the present application, in order to obtain accurate bifurcation point marking, different calculation methods are sampled to alternately calculate the lane line bifurcation point error to achieve lane line bifurcation point correction. Specifically, the lane line bifurcation point error is calculated using the first method and the second method described in the above embodiment. For example, for the i-th correction, the lane line bifurcation point error is calculated using the first method, and the lane line bifurcation point error is calculated using the second method for the i+1th correction. Sequentially, the lane line bifurcation point error is calculated using the first method for the i+2th correction.

[0104] In an optional embodiment, the error between the predicted lane line point set and the projected lane line point set is calculated by adding a lane line smoothness constraint to correct the smoothness of the 3D lane line annotation point set. Specifically, the error between the predicted lane line point set and the projected lane line point set is obtained according to the following steps B1 to B3:

[0105] Step B1: Determine the smoothness corresponding to each projection lane line point in the projection lane line point set according to each projection lane line point in the projection lane line point set and two adjacent projection lane line points.

[0106] Step B2: Determine the weight corresponding to each projection lane line point in the projection lane line point set according to the smoothness corresponding to each projection lane line point in the projection lane line point set.

[0107] Step B3: Determine the error between the predicted lane line point set and the projected lane line point set based on the error and weight corresponding to each projected lane line point in the projected lane line point set.

[0108] In the embodiment of the present application, the smoothness constraint requires that the directions between adjacent points on the projected lane line point set must remain consistent. For example, for three projected lane line points, a direction is determined between the first projected lane line point and the second projected lane line point, and a direction is also determined between the second projected lane line point and the third projected lane line point. The smoothness constraint requires that these two directions remain consistent. Therefore, in step B1, by calculating the direction between each projected lane line point and two adjacent projected lane line points, the smoothness of each projected lane line point can be determined based on the direction. Specifically, the consistent the direction, the smoother the projected lane line point; if the direction difference is large, it means that the projected lane line point is not smooth.

[0109] At the same time, smoothness can reflect the accuracy of the 3D lane line annotation point set. The higher the smoothness, the more accurate the 3D lane line annotation point corresponding to the projected lane line point; and the worse the smoothness, the less accurate the 3D lane line annotation point corresponding to the projected lane line point. Such inaccurate 3D lane line annotation points need to be corrected. Therefore, in step B2, a corresponding weight is assigned to each projected lane line point according to the smoothness of the projected lane line point, that is, a small weight is assigned to a projected lane line point with high smoothness, and a large weight is assigned to a projected lane line point with poor smoothness.

[0110] Step B3 calculates the error based on the weight corresponding to each projected lane line point, and adds smoothness to the error. Then, when the 3D lane line annotation point set is subsequently corrected based on the error between the predicted lane line point set and the projected lane line point set, the unsmooth 3D lane line annotation points are also corrected, solving the problems of lane line disorder and unevenness, and obtaining a corrected 3D lane line annotation point set with higher accuracy.

[0111] In an optional embodiment, projecting the reference point set into the target image, and / or projecting the 3D lane line annotation point set into the target image, is performed according to a current projection method corresponding to the current posture. Since the posture collected by the inertial navigation sensor of the vehicle has a certain error, which leads to an inaccurate projection method, it is necessary to correct the posture to obtain an accurate projection method.

[0112] Specifically, the method includes: correcting the current frame posture to obtain a corrected posture; projecting the reference point set into the target image according to the corrected projection method corresponding to the corrected posture, and / or projecting the 3D lane line annotation point set into the target image.

[0113] In the embodiment of the present application, the posture is corrected to obtain the corrected posture, and the corresponding camera coordinate system is determined according to the corrected posture, and then the corresponding projection method is determined. The accuracy of the projection method is guaranteed, and then the projection is performed according to the corrected projection method, which ensures the accuracy of the projection of the 3D lane line annotation point set and the reference point set, and avoids the projection error caused by the inaccurate projection method.

[0114] In an optional embodiment, after obtaining the 3D lane line annotation point set, the 3D lane line annotation point set is spliced ​​with the local 3D lane line annotation point set (i.e., the 3D lane line annotation point set before the current moment), and the 3D lane lines in the 3D lane line annotation point set are matched with the 3D lane lines in the local 3D lane line annotation point set through the km algorithm to obtain a larger expanded 3D lane line annotation point set. Then, the expanded 3D lane line annotation point set is directly corrected in the subsequent steps.

[0115] Figure 6 It is a schematic diagram of the overall framework of a 3D lane line marking method provided in an embodiment of the present application. It mainly includes three stages: map data preprocessing, 3D lane line marking point set correction, and post-processing and data acceptance. In the map data preprocessing stage, according to the current posture, the 3D lane line marking point set matching the current posture is extracted from the local 3D map associated with the target image, and the lane lines of the target image are predicted to obtain the predicted lane line point set; the 3D lane line marking point set is projected into the target image to obtain the projected lane line point set, and the association relationship between the projected lane lines and the predicted lane lines is established. In the 3D lane line annotation point set correction stage, the pixel point set used to represent the same reference point is determined from the target image and the first N lane line images of the target image, and the 3D position of the reference point is determined (i.e., triangulation) based on the pixel point set, the camera pose when acquiring the target image and the first N lane line images. The local 3D lane line annotation point set and the 3D lane line annotation point set are spliced ​​to obtain the augmented 3D lane line annotation point set; then the augmented 3D lane line annotation point set is corrected based on the reference point set and the predicted lane line point set to obtain the corrected 3D lane line annotation point set. Finally, in the post-processing and data acceptance stage, the corrected 3D lane line annotation point set is evaluated based on the predicted lane line point set, and the 3D line type of the corrected 3D lane line annotation point set is evaluated. The data is cleaned by comprehensively considering the two evaluation results, and training samples are generated based on the cleaned 3D lane line annotation point set.

[0116] The present application also provides a 3D lane marking device. Figure 7 As shown, Figure 7 700 is a schematic diagram of a 3D lane marking device provided in an embodiment of the present application. The device 700 includes:

[0117] An acquisition module 710 is used to acquire a target image and a current posture, wherein the target image is an image including lane lines, and the current posture is a posture of the camera when acquiring the target image;

[0118] An extraction module 720 is used to extract a 3D lane line annotation point set matching the current posture from a local 3D map associated with the target image according to the current posture;

[0119] A prediction module 730 is used to perform lane line prediction on the target image to obtain a predicted lane line point set;

[0120] A projection module 740 is used to project the 3D lane line annotation point set into the target image to obtain a projected lane line point set;

[0121] The correction module 750 is used to correct the 3D lane line annotation point set according to the error between the projected lane line point set and the predicted lane line point set to obtain a corrected 3D lane line annotation point set.

[0122] In an optional embodiment, the correction module includes:

[0123] A first correction submodule, configured to correct the 3D lane line annotation point set at least according to a lane line reprojection error between the projected lane line point set and the predicted lane line point set;

[0124] The lane line reprojection error is used to characterize the error between every two matching point pairs in the projected lane line point set and the predicted lane line point set.

[0125] In an optional embodiment, the correction module includes:

[0126] A second correction submodule, configured to correct the 3D lane line annotation point set according to at least one of a lane line segmentation point error and a lane line bifurcation point error between the projected lane line point set and the predicted lane line point set, and the lane line reprojection error;

[0127] The lane line segmentation point error is used to characterize the error between the projected lane line segmentation point in the projected lane line point set and the predicted lane line segmentation point in the predicted lane line point set;

[0128] The lane line bifurcation point error is used to characterize the error between the projected lane line bifurcation point in the projected lane line point set and the predicted lane line bifurcation point in the predicted lane line point set.

[0129] In an optional embodiment, the device further includes:

[0130] A first confirmation module is used to determine, for each projected lane line point in the projected lane line point set, a predicted lane line point that is closest to the projected lane line point from the predicted lane line point set;

[0131] The first error module is used to determine the lane line reprojection error according to the distance between each projected lane line point in the projected lane line point set and the nearest predicted lane line point.

[0132] In an optional embodiment, the device further includes:

[0133] A second confirmation module is used to determine a matching predicted lane line segmentation point from the predicted lane line point set for each projected lane line segmentation point in the projected lane line point set, wherein the matched projected lane line segmentation point and the predicted lane line segmentation point are used to represent the same segmentation point on the lane line;

[0134] The second error module is used to determine the lane line segmentation point error according to the distance between each matched projected lane line segmentation point and the predicted lane line segmentation point.

[0135] In an optional embodiment, the device further includes:

[0136] A third confirmation module is used to determine a matching predicted lane line bifurcation point from the predicted lane line point set for each projected lane line bifurcation point in the projected lane line point set, wherein the matching projected lane line bifurcation point and the predicted lane line bifurcation point are used to represent the same bifurcation point on the lane line;

[0137] The third error module is used to determine the lane line bifurcation point error between each matched projected lane line bifurcation point and the predicted lane line bifurcation point.

[0138] In an optional embodiment, the 3D lane line annotation point set is corrected multiple times; the correction module includes:

[0139] A third correction submodule is used to obtain a corrected 3D lane line annotation point set after performing a correction on the 3D lane line annotation point set each time;

[0140] The fourth correction submodule is used to project the 3D lane line annotation point set after each correction into the target image to obtain the projected lane line point set after this correction;

[0141] A fifth correction submodule, configured to continue the next correction when the error between the projected lane line point set after the current correction and the predicted lane line point set does not meet the target condition;

[0142] The sixth correction submodule is used to stop the correction when the error between the projected lane line point set after the current correction and the predicted lane line point set meets the target condition, so as to obtain the 3D lane line annotation point set after the last correction.

[0143] In an optional embodiment, the correction module further includes an error calculation module, and the error calculation module includes:

[0144] A smoothness module, used to determine the smoothness corresponding to each projection lane line point in the projection lane line point set according to each projection lane line point and two adjacent projection lane line points in the projection lane line point set;

[0145] A weight module, used to determine a weight corresponding to each projection lane line point in the projection lane line point set according to a smoothness corresponding to each projection lane line point in the projection lane line point set;

[0146] The error determination module is used to determine the error between the predicted lane line point set and the projected lane line point set according to the error and weight corresponding to each projected lane line point in the projected lane line point set.

[0147] In an optional embodiment, the device further includes:

[0148] A reference point acquisition module, used to acquire a reference point set, wherein the 3D position of each reference point in the reference point set is predetermined, and each reference point in the reference point set is a pixel point used to characterize a reference object in the target image;

[0149] A reference point projection module, used for projecting the reference point set into the target image to obtain a projected reference point set;

[0150] A pixel point determination module, used to determine a pixel point set representing each reference point from the target image;

[0151] The second correction module is used to correct the 3D lane line annotation point set according to the error between the projected lane line point set and the predicted lane line point set, and the error between the projected reference point and the pixel point set.

[0152] In an optional embodiment, the device further includes:

[0153] A distance calculation module, for calculating, for each projected lane line formed by the projected lane line point set, the distance between a point on the projected lane line and each predicted lane line formed by the predicted lane line point set;

[0154] An association module, used for determining the associated projected lane line and predicted lane line according to the calculated distance;

[0155] The fourth correction module is used to correct the 3D lane line annotation point set according to the error between the projected lane line point set of the associated projected lane line and the predicted lane line point set of the predicted lane line to obtain a corrected 3D lane line annotation point set.

[0156] The present application also provides an electronic device, referring to Figure 8 , Figure 8Schematic diagram of an electronic device structure provided by an embodiment of the present application. Figure 8 As shown, the electronic device 800 includes: a memory 810 and a processor 820. The memory 810 and the processor 820 are connected via a bus communication. A computer program is stored in the memory 810. The computer program can be run on the processor 820 to implement the steps of the 3D lane line marking method described in the embodiment of the present application.

[0157] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the 3D lane line marking method described in the embodiment of the present application are implemented.

[0158] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the 3D lane line marking method described in the embodiment of the present application.

[0159] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0160] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0163] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0164] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0165] The above is a detailed introduction to a 3D lane line marking method, electronic device and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A 3D lane marking method, characterized in that: The method comprises: Acquire a target image, and acquire a current posture, wherein the target image is an image including lane lines, and the current posture is the posture of the camera when acquiring the target image; According to the current posture, extracting a 3D lane line annotation point set matching the current posture from a local 3D map associated with the target image; Performing lane line prediction on the target image to obtain a predicted lane line point set; Projecting the 3D lane line annotation point set into the target image to obtain a projected lane line point set; According to the error between the projected lane line point set and the predicted lane line point set, the 3D lane line annotation point set is corrected to obtain a corrected 3D lane line annotation point set.

2. The method according to claim 1, characterized in that According to the error between the projected lane line point set and the predicted lane line point set, the 3D lane line annotation point set is corrected, including: Correcting the 3D lane line annotation point set at least according to a lane line reprojection error between the projected lane line point set and the predicted lane line point set; The lane line reprojection error is used to characterize the error between every two matching point pairs in the projected lane line point set and the predicted lane line point set.

3. The method according to claim 2, characterized in that The 3D lane line annotation point set is corrected based on at least a lane line reprojection error between the projected lane line point set and the predicted lane line point set, including: Correcting the 3D lane line annotation point set according to at least one of a lane line segmentation point error and a lane line bifurcation point error between the projected lane line point set and the predicted lane line point set, and the lane line reprojection error; The lane line segmentation point error is used to characterize the error between the projected lane line segmentation point in the projected lane line point set and the predicted lane line segmentation point in the predicted lane line point set; The lane line bifurcation point error is used to characterize the error between the projected lane line bifurcation point in the projected lane line point set and the predicted lane line bifurcation point in the predicted lane line point set.

4. The method according to claim 2 or 3, characterized in that: The method further comprises: For each projected lane line point in the projected lane line point set, determining a predicted lane line point that is closest to the projected lane line point from the predicted lane line point set; The lane line reprojection error is determined based on the distance between each projected lane line point in the projected lane line point set and the nearest predicted lane line point.

5. The method according to claim 3, characterized in that: The method further comprises: For each projected lane line segmentation point in the projected lane line point set, determining a matching predicted lane line segmentation point from the predicted lane line point set, wherein the matching projected lane line segmentation point and the predicted lane line segmentation point are used to represent the same segmentation point on the lane line; The lane line segmentation point error is determined based on the distance between each matched projected lane line segmentation point and the predicted lane line segmentation point.

6. The method according to claim 3, characterized in that The method further comprises: For each projected lane line bifurcation point in the projected lane line point set, determining a matching predicted lane line bifurcation point from the predicted lane line point set, wherein the matching projected lane line bifurcation point and the predicted lane line bifurcation point are used to represent the same bifurcation point on the lane line; The lane line bifurcation point error between each matched projected lane line bifurcation point and the predicted lane line bifurcation point is determined.

7. The method according to any one of claims 1 to 6, characterized in that: The 3D lane line annotation point set is corrected multiple times; the 3D lane line annotation point set is corrected according to the error between the predicted lane line point set and the projected lane line point set to obtain a corrected 3D lane line annotation point set, including: After each correction is performed on the 3D lane line annotation point set, a corrected 3D lane line annotation point set is obtained; Projecting each corrected 3D lane line annotation point set onto the target image to obtain the corrected projected lane line point set; When the error between the projected lane line point set after this correction and the predicted lane line point set does not meet the target condition, proceed to the next correction; When the error between the projected lane line point set after this correction and the predicted lane line point set meets the target condition, the correction is stopped to obtain the 3D lane line annotation point set after the last correction.

8. The method according to any one of claims 1 to 7, characterized in that: The error between the predicted lane line point set and the projected lane line point set is obtained according to the following steps: Determining the smoothness corresponding to each projection lane line point in the projection lane line point set according to each projection lane line point in the projection lane line point set and two adjacent projection lane line points; Determining a weight corresponding to each projected lane line point in the projected lane line point set according to a smoothness corresponding to each projected lane line point in the projected lane line point set; The error between the predicted lane line point set and the projected lane line point set is determined according to the error and weight corresponding to each projected lane line point in the projected lane line point set.

9. The method according to any one of claims 1 to 8, characterized in that: After acquiring the target image and the current posture, the method further includes: Acquire a reference point set, wherein the 3D position of each reference point in the reference point set is predetermined, and each reference point in the reference point set is a pixel point used to characterize a reference object in the target image; Projecting the reference point set into the target image to obtain a projected reference point set; Determining, from the target image, a set of pixel points representing each of the reference points; According to the error between the projected lane line point set and the predicted lane line point set, the 3D lane line annotation point set is corrected to obtain a corrected 3D lane line annotation point set, including: The 3D lane line annotation point set is corrected according to the error between the projected lane line point set and the predicted lane line point set, and the error between the projected reference point and the pixel point set.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: For each projected lane line formed by the projected lane line point set, calculating the distance between a point on the projected lane line and each predicted lane line formed by the predicted lane line point set; Determine the associated projected lane line and predicted lane line according to the calculated distance; According to the error between the projected lane line point set and the predicted lane line point set, the 3D lane line annotation point set is corrected to obtain a corrected 3D lane line annotation point set, including: According to the error between the projected lane line point set of the associated projected lane line and the predicted lane line point set of the predicted lane line, the 3D lane line annotation point set is corrected to obtain a corrected 3D lane line annotation point set.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the 3D lane line marking method according to any one of claims 1 to 10 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the 3D lane line marking method described in any one of claims 1-10 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the 3D lane marking method as described in any one of claims 1 to 10 are implemented.

Images