Three-dimensional lane line generation method and device, electronic equipment and computer readable medium

Abstract

Embodiments of the present disclosure disclose a three-dimensional lane line generation method, device, electronic equipment and computer readable medium. A specific implementation of the method comprises: performing curve fitting on each of the post-projection lane line sampling point coordinates in each post-projection lane line sampling point coordinate group in the post-projection lane line sampling point coordinate group sequence to generate a fitted lane line, thereby obtaining a fitted lane line sequence; performing adjustment processing on the coordinate system conversion matrix to generate a target conversion matrix; performing coordinate projection on each of the lane line sampling point coordinates in each lane line sampling point coordinate group in the lane line sampling point coordinate group sequence to generate a target lane line sampling point coordinate group, thereby obtaining a target lane line sampling point coordinate group sequence; and performing lane line fitting on each of the target lane line sampling points in each target lane line sampling point coordinate group in the target lane line sampling point coordinate group sequence, thereby obtaining a three-dimensional lane line equation set. The implementation can improve the accuracy of the generated three-dimensional lane line.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to a method, apparatus, electronic device, and computer-readable medium for generating three-dimensional lane lines. Background Technology

[0002] 3D lane line generation is an important technology in the field of autonomous driving. Currently, the common approach to generating 3D lane lines is as follows: First, feature matching is performed on lane line corner points across multiple road images to solve for the camera's position and pose matrix relative to the ground. Then, using the camera's position and pose matrix relative to the ground, the extracted lane line corner points are projected onto the vehicle coordinate system to generate the 3D lane line.

[0003] However, the inventors discovered that when generating obstacle information using the above method, the following technical problems often arise:

[0004] First, if the corner points of the lane lines cannot be extracted from the road image, it is difficult to solve the position and pose matrix of the camera relative to the ground, thus making it impossible to generate three-dimensional lane lines.

[0005] Second, if lane corner points cannot be extracted from the road image, even if the lane corner points of the lane lines in the pre-stored high-precision map are used as the lane corner points of the road image and projected onto the vehicle coordinate system to generate three-dimensional lane lines, the accuracy of the three-dimensional coordinates in the vehicle coordinate system will be insufficient because the high-precision map does not contain elevation information (i.e., height values). As a result, the accuracy of the generated three-dimensional lane lines will be reduced.

[0006] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0008] Some embodiments of this disclosure provide methods, apparatuses, electronic devices, and computer-readable media for generating three-dimensional lane lines to address one or more of the technical problems mentioned in the background section above.

[0009] In a first aspect, some embodiments of this disclosure provide a method for generating three-dimensional lane lines. The method includes: extracting sampling points from a pre-acquired road image to obtain a sequence of lane line sampling point coordinates; based on a preset camera pose matrix, a coordinate system transformation matrix, and pre-acquired current vehicle coordinate values, projecting the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the above-mentioned lane line sampling point coordinate sequence to generate a projected lane line sampling point coordinate group, thus obtaining a sequence of projected lane line sampling point coordinates; and performing curve fitting on the coordinates of each projected lane line sampling point in each projected lane line sampling point coordinate group in the above-mentioned sequence of projected lane line sampling point coordinates to generate a fitted lane line, thus obtaining... Fit a lane line sequence; based on the fitted lane line sequence and the map lane line information sequence that matches the current vehicle coordinate values ​​included in the preset high-precision map data, adjust the coordinate system transformation matrix to generate a target transformation matrix; using the target transformation matrix, project the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the lane line sampling point coordinate group sequence to generate a target lane line sampling point coordinate group, thus obtaining a target lane line sampling point coordinate group sequence; fit lane lines to each target lane line sampling point in each target lane line sampling point in each target lane line sampling point coordinate group sequence to generate a three-dimensional lane line equation, thus obtaining a three-dimensional lane line equation set.

[0010] Secondly, some embodiments of this disclosure provide a three-dimensional lane line generation apparatus, which includes: an extraction unit configured to extract sampling points from a pre-acquired road image to obtain a sequence of lane line sampling point coordinate sets; a first coordinate projection unit configured to perform coordinate projection on the coordinates of each lane line sampling point in each lane line sampling point coordinate set in the above-mentioned lane line sampling point coordinate set sequence based on a preset camera pose matrix, a coordinate system transformation matrix, and pre-acquired current vehicle coordinate values ​​to generate a projected lane line sampling point coordinate set sequence; and a first fitting unit configured to perform curve fitting on the coordinates of each projected lane line sampling point in each projected lane line sampling point coordinate set in the above-mentioned projected lane line sampling point coordinate set sequence to generate a fitted lane line, obtaining a fitted lane line. The system comprises: a lane line sequence fitting unit; an adjustment processing unit configured to adjust the coordinate system transformation matrix based on the fitted lane line sequence and a map lane line information sequence in preset high-precision map data that matches the current vehicle coordinate values, to generate a target transformation matrix; a second coordinate projection unit configured to use the target transformation matrix to project the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the lane line sampling point coordinate group sequence to generate a target lane line sampling point coordinate group, thus obtaining a target lane line sampling point coordinate group sequence; and a second fitting unit configured to perform lane line fitting on each target lane line sampling point in each target lane line sampling point in each target lane line sampling point coordinate group in the target lane line sampling point coordinate group sequence to generate a three-dimensional lane line equation, thus obtaining a three-dimensional lane line equation set.

[0011] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0012] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0013] Fifthly, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0014] The above-described embodiments of this disclosure have the following beneficial effects: The three-dimensional lane line generation method of some embodiments of this disclosure can generate three-dimensional lane lines even when lane line corner points extracted from road images are missing. Specifically, the reason why three-dimensional lane lines cannot be generated is that if lane line corner points cannot be extracted from the road image, it is difficult to solve the position and pose matrix of the camera relative to the ground. Based on this, the three-dimensional lane line generation method of some embodiments of this disclosure first extracts sampling points from the pre-acquired road image to obtain a sequence of lane line sampling point coordinates. Here, lane line sampling points can be coordinates with different features from lane line corner points, and both can be used to characterize the features of the lane line. Thus, the lane line sampling point coordinates obtained through sampling point extraction can be used to replace the lane line corner points that cannot be extracted for three-dimensional lane line generation. Second, based on a preset camera pose matrix, coordinate system transformation matrix, and pre-acquired current vehicle coordinate values, coordinate projection is performed on the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the above-described lane line sampling point coordinate group sequence to generate a projected lane line sampling point coordinate group, resulting in a sequence of projected lane line sampling point coordinate groups. Coordinate projection allows for the initial determination of the 3D coordinates corresponding to the extracted lane line corner points, i.e., the coordinates of the projected lane line sampling points. Then, curve fitting is performed on the coordinates of each projected lane line sampling point in each of the projected lane line sampling point coordinate groups to generate fitted lane lines, resulting in a fitted lane line sequence. Curve fitting can be used to fit 3D lane line equations in the vehicle coordinate system. Subsequently, based on the fitted lane line sequence and the map lane line information sequence from the preset high-precision map data that matches the current vehicle coordinate values, the coordinate system transformation matrix is ​​adjusted to generate the target transformation matrix. By introducing the map lane line information sequence from the high-precision map data, the coordinate system transformation matrix can be adjusted, thereby improving the accuracy of the target transformation matrix. Because the accuracy of the target transformation matrix is ​​improved, it can be used to characterize the pose matrix of the camera coordinate system relative to the ground plane coordinate system. Thus, a more accurate pose matrix is ​​obtained without relying on lane line corner points. Next, using the aforementioned target transformation matrix, coordinate projection is performed on the coordinates of each lane line sampling point in each lane line sampling point coordinate group sequence to generate the target lane line sampling point coordinate group, resulting in the target lane line sampling point coordinate group sequence. Here, provided the accuracy of the target transformation matrix is ​​ensured, further coordinate projection can improve the accuracy of the target lane line sampling point coordinates. Finally, lane line fitting is performed on each target lane line sampling point in each target lane line sampling point coordinate group sequence to generate a three-dimensional lane line equation, resulting in a three-dimensional lane line equation set.Therefore, it is possible to generate 3D lane lines without relying on lane line corner points, and to improve the accuracy of the generated 3D lane lines. Attached Figure Description

[0015] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0016] Figure 1 This is a flowchart of some embodiments of the three-dimensional lane line generation method according to the present disclosure;

[0017] Figure 2 These are schematic diagrams illustrating the structure of some embodiments of the three-dimensional lane line generation apparatus according to this disclosure;

[0018] Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0019] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0020] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0021] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0022] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0023] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0024] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] Figure 1 A flow 100 of some embodiments of the three-dimensional lane line generation method according to the present disclosure is shown. The three-dimensional lane line generation method includes the following steps:

[0026] Step 101: Extract sampling points from the pre-acquired road image to obtain a sequence of lane line sampling point coordinates.

[0027] In some embodiments, the execution entity of the 3D lane line generation method can extract sampling points from a pre-acquired road image to obtain a sequence of lane line sampling point coordinates. The road image can be captured by an onboard camera of the current vehicle. A preset sampling point extraction algorithm can be used to extract sampling points from the road image to obtain the sequence of lane line sampling point coordinates. Here, each lane line sampling point coordinate group in the sequence represents a lane line in the road image. Furthermore, there is an order among the lane line sampling point coordinate groups in the sequence. Additionally, the lane line sampling point coordinates are two-dimensional coordinates in the image coordinate system.

[0028] As an example, the coordinate groups of each lane line sampling point in the lane line sampling point coordinate group sequence can be arranged in order from left to right for the corresponding lane line. The above sampling point extraction algorithm may include, but is not limited to, at least one of the following: SIFT (Scale-invariant Feature Transform) algorithm, Surf (SpeededUp Robust Features) algorithm, Harris corner detection, FAST corner detection, and BRIEF (BinaryRobust Independent Elementary Features) algorithm.

[0029] Step 102: Based on the preset camera pose matrix, coordinate system transformation matrix and pre-acquired current vehicle coordinate values, perform coordinate projection on the coordinates of each lane line sampling point in each lane line sampling point coordinate group sequence to generate the projected lane line sampling point coordinate group, thus obtaining the projected lane line sampling point coordinate group sequence.

[0030] In some embodiments, the aforementioned execution entity may, based on a preset camera pose matrix, coordinate system transformation matrix, and pre-acquired current vehicle coordinate values, perform coordinate projection on the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the aforementioned lane line sampling point coordinate group sequence in various ways to generate a projected lane line sampling point coordinate group, thereby obtaining a projected lane line sampling point coordinate group sequence.

[0031] In some optional implementations of certain embodiments, the execution entity, based on a preset camera pose matrix, coordinate system transformation matrix, and pre-acquired current vehicle coordinate values, performs coordinate projection on the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the above lane line sampling point coordinate group sequence to generate a projected lane line sampling point coordinate group, thereby obtaining a projected lane line sampling point coordinate group sequence. This may include the following steps:

[0032] The first step is to construct a ground-plane coordinate system using the current vehicle coordinates and the pre-acquired camera height. Specifically, the origin of the ground-plane coordinate system can be defined as the distance along the vertical axis of the current vehicle coordinates reduced by the distance of the camera height. The horizontal axis, passing through the origin and perpendicular to the vehicle's direction of travel, is used as the horizontal axis. The vertical axis, passing through the origin and perpendicular to both the horizontal and vertical axes, is used as the vertical axis. Secondly, the camera coordinate system of the vehicle-mounted camera can be vertically shifted downwards by the distance of the camera height to obtain the ground-plane coordinate system.

[0033] The second step involves projecting the coordinates of each lane line sampling point in each lane line sampling point coordinate group within the aforementioned lane line sampling point coordinate group sequence onto the aforementioned ground plane coordinate system to generate a projected lane line sampling point coordinate group sequence. The camera pose matrix can be used to characterize the position and orientation of the vehicle-mounted camera when capturing the road image. The coordinate system transformation matrix can be a transformation matrix between the aforementioned ground plane coordinate system and the camera coordinate system. Therefore, the lane line sampling point coordinates can be projected onto the aforementioned ground plane coordinate system to generate the projected lane line sampling point coordinates. The following formula can be used to project the coordinates of each lane line sampling point in each lane line sampling point coordinate group within the aforementioned lane line sampling point coordinate group sequence onto the aforementioned ground plane coordinate system to generate the projected lane line sampling point coordinates:

[0034]

[0035] Where K represents the camera intrinsic parameter matrix of the aforementioned vehicle-mounted camera. H represents the relative pose matrix between coordinate systems. This represents the pose matrix between the ground plane coordinate system and the corresponding camera coordinate system. u represents the abscissa value of the lane line sampling point. v represents the ordinate value of the lane line sampling point. x represents the abscissa value of the lane line sampling point after projection. y represents the ordinate value of the lane line sampling point after projection. Z represents the ordinate value of the lane line sampling point after projection. λ represents the balancing coefficient, used to make both sides of the equation equal. n represents the ground plane normal vector. T represents the transpose of the matrix.

[0036] Step 103: Perform curve fitting on the coordinates of each projected lane line sampling point in each projected lane line sampling point coordinate group sequence to generate a fitted lane line, thus obtaining a fitted lane line sequence.

[0037] In some embodiments, the execution entity may perform curve fitting on the coordinates of each projected lane line sampling point in each projected lane line sampling point coordinate group in the above-mentioned sequence of projected lane line sampling point coordinate groups to generate a fitted lane line, thereby obtaining a fitted lane line sequence. The fitted lane line in the fitted lane line sequence may be a three-dimensional lane line in the above-mentioned ground plane coordinate system.

[0038] Step 104: Based on the fitted lane line sequence and the map lane line information sequence that matches the current vehicle coordinate value included in the preset high-precision map data, the coordinate system transformation matrix is ​​adjusted to generate the target transformation matrix.

[0039] In some embodiments, the execution entity may adjust the coordinate system transformation matrix in various ways based on the fitted lane line sequence and the map lane line information sequence in the preset high-precision map data that matches the current vehicle coordinate value, so as to generate a target transformation matrix.

[0040] In some optional implementations of certain embodiments, the execution entity adjusts the coordinate system transformation matrix based on the fitted lane line sequence and the map lane line information sequence in the preset high-precision map data that matches the current vehicle coordinate values, in order to generate a target transformation matrix. This may include the following steps:

[0041] The first step involves generating a lane width error value sequence and a lane curvature error value sequence based on the fitted lane line sequence and the map lane line information sequence in the preset high-precision map data that matches the current vehicle coordinates. The high-precision map data can be the map data showing the current vehicle's location within a high-precision map. This preset high-precision map data can be extracted from the high-precision map using the current vehicle coordinates.

[0042] The second step involves adjusting the coordinate system transformation matrix based on the lane width error value sequence and the lane curvature error value sequence to generate the target transformation matrix.

[0043] In some optional implementations of certain embodiments, each map lane line information in the above-mentioned map lane line information sequence may include a map lane line. The execution entity generates a lane line width error value sequence and a lane line curvature error value sequence based on the above-mentioned fitted lane line sequence and the map lane line information sequence in the preset high-precision map data that matches the above-mentioned current vehicle coordinate values. This may include the following steps:

[0044] The first step is to determine the fitted lane width value between every two adjacent fitted lane lines in the above fitted lane line sequence, thus obtaining a fitted lane line width value sequence. The fitted lane line width value between two adjacent fitted lane lines can be used to characterize the width between two adjacent lane lines in the road image. Here, the fitted lane line width value can be determined from the above-mentioned ground plane coordinate system. Each piece of map lane line information in the above map lane line information sequence may also include, but is not limited to, at least one of the following: map lane line identifier, lane line color, lane line type (solid line, dashed line), etc.

[0045] The second step is to determine the lane width value between each pair of adjacent lane line information in the above map lane line information sequence, thus obtaining a map lane line width value sequence. Here, since the map lane line information sequence is lane line information from a high-precision map, the lane width value between each pair of adjacent lane line information can also be a lane line width value with high accuracy.

[0046] The third step involves determining the difference between each fitted lane width value and its corresponding map lane width value in the aforementioned fitted lane width value sequence as the lane width error value, thus obtaining the lane width error value sequence. Each fitted lane line in the aforementioned fitted lane line sequence can correspond to a fitted lane line identifier, used to uniquely identify a lane line. Here, "corresponding" means that the fitted lane line identifiers of the two fitted lane lines corresponding to the fitted lane width values ​​are the same as the map lane line identifiers of the map lane lines corresponding to the map lane width values.

[0047] The fourth step is to determine the curvature value of each fitted lane line in the above fitted lane line sequence, thus obtaining a sequence of fitted lane line curvature values. The curvature value of each fitted lane line in the above fitted lane line sequence can be determined using a curvature formula.

[0048] The fifth step is to determine the map lane curvature value of each map lane information included in the above map lane information sequence, and obtain the map lane curvature value sequence.

[0049] The sixth step involves determining the difference between each fitted lane curvature value and its corresponding map lane curvature value in the aforementioned sequence of fitted lane width values ​​as the lane curvature error value, thus obtaining a sequence of lane curvature error values. Here, "corresponding" can refer to either the fitted lane label of the fitted lane line corresponding to the fitted lane curvature value or the map lane label of the map lane line corresponding to the map lane curvature value.

[0050] In some optional implementations of certain embodiments, the execution entity adjusts the coordinate system transformation matrix based on the lane width error value sequence and the lane curvature error value sequence to generate the target transformation matrix, which may include the following steps:

[0051] The first step is to obtain the preset weight coefficients and the camera relative pose matrix. The weight coefficients can be coefficients for the error values. The camera relative pose matrix can be the position and orientation matrix of the camera coordinate system relative to the vehicle coordinate system of the current vehicle. Here, the camera relative pose matrix can be pre-calibrated.

[0052] As an example, the weighting coefficient can be a number in the range [0.5, 1]. For example, it can be 0.9.

[0053] The second step involves decomposing the aforementioned coordinate system transformation matrix to generate the vehicle's roll and pitch angles. First, the coordinate system transformation matrix can be converted into the inverse of the product of the transformation matrix of the ground plane coordinate system relative to the vehicle coordinate system and the transformation matrix of the vehicle coordinate system relative to the camera coordinate system. Here, the transformation matrix of the ground plane coordinate system relative to the vehicle coordinate system can include the rotation matrix and translation vector of the ground plane coordinate system relative to the vehicle coordinate system. The vertical coordinate value of this translation vector can be determined as the decomposed height value to characterize the current vehicle's height relative to the ground. Second, the rotation matrix can be decomposed using Euler angle decomposition to obtain the vehicle roll and pitch angles.

[0054] Thirdly, based on the aforementioned lane width error value sequence, lane curvature error value sequence, weighting coefficients, and camera relative pose matrix, the aforementioned vehicle roll angle, vehicle pitch angle, and camera height values ​​are adjusted to generate a target transformation matrix. This target transformation matrix can be a position and attitude matrix of the ground plane coordinate system relative to the camera coordinate system. Furthermore, the target transformation matrix and target height values ​​can be generated using the following formulas:

[0055]

[0056] in, The target transformation matrix is ​​represented by , which is the position and attitude matrix of the camera coordinate system relative to the ground plane coordinate system. `l` represents the overall loss value of the lane lines. `α` represents the weighting coefficients. `i` and `j` represent the indices. `e` represents the error value. d This represents the lane width error value in the above sequence of lane width error values. d，j This represents the j-th lane width error value in the above sequence of lane width error values. c This represents the lane curvature error value in the above sequence of lane curvature error values. c，i This represents the i-th lane curvature error value in the above sequence of lane curvature error values. arg min() is the objective function to minimize.

[0057] Here, minimizing the objective function can be used to ensure that the position and attitude matrix of the camera coordinate system relative to the ground plane coordinate system is minimized after adjusting the vehicle roll angle, vehicle pitch angle, and camera height values, thus obtaining the target transformation matrix. The target height value can be obtained from the target transformation matrix using the decomposition process described above.

[0058] Step 105: Using the target transformation matrix, project the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the lane line sampling point coordinate group sequence to generate the target lane line sampling point coordinate group, thus obtaining the target lane line sampling point coordinate group sequence.

[0059] In some embodiments, the execution entity may use the target transformation matrix to project the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the lane line sampling point coordinate group sequence in various ways to generate a target lane line sampling point coordinate group, thereby obtaining a target lane line sampling point coordinate group sequence.

[0060] In some optional implementations of certain embodiments, the execution entity utilizes the target transformation matrix to perform coordinate projection on the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the lane line sampling point coordinate group sequence to generate a target lane line sampling point coordinate group, thereby obtaining a target lane line sampling point coordinate group sequence. This may include the following steps:

[0061] The first step is to update the ground plane coordinate system based on the target height value to obtain the target ground plane coordinate system. Specifically, the camera coordinate system can be shifted downwards along the vertical axis by the distance of the target height value to obtain the target ground plane coordinate system, thus completing the update of the ground plane coordinate system.

[0062] The second step involves using the inverse of the aforementioned target transformation matrix to project the coordinates of each lane line sampling point in each lane line sampling point coordinate group sequence to the aforementioned target ground plane coordinate system, thereby generating the target lane line sampling point coordinate group sequence. Since the aforementioned target transformation matrix is ​​the position and attitude matrix of the ground plane coordinate system relative to the camera coordinate system, its inverse can be used as the coordinate transformation matrix to obtain the transformation matrix of the camera coordinate system relative to the ground plane coordinate system. This transformation can then be used to transform the lane line sampling point coordinates to the target ground plane coordinate system. Furthermore, the aforementioned coordinate projection formula can be used to project the coordinates of each lane line sampling point in each lane line sampling point coordinate group sequence to the aforementioned target ground plane coordinate system, thereby generating the target lane line sampling point coordinate group sequence.

[0063] The above formulas and related contents, as an inventive point of this disclosure, solve the second technical problem mentioned in the background art: "When lane corner points cannot be extracted from road images, even if lane corner points from pre-stored high-precision maps are used as lane corner points of road images projected onto the vehicle coordinate system to generate 3D lane lines, the accuracy of the 3D coordinates in the vehicle coordinate system will be insufficient due to the lack of elevation information (i.e., height values) in the high-precision map, thus reducing the accuracy of the generated 3D lane lines." The reason for the reduced accuracy of the generated 3D lane lines is that, when lane corner points cannot be extracted from road images, even if lane corner points from pre-stored high-precision maps are used as lane corner points of road images projected onto the vehicle coordinate system to generate 3D lane lines, the accuracy of the 3D coordinates in the vehicle coordinate system will be insufficient due to the lack of elevation information (i.e., height values) in the high-precision map. If the above factors are solved, the accuracy of the generated 3D lane lines can be improved. To achieve this effect, firstly, when lane corner point features are missing in the road image, pre-stored high-precision map data is introduced. Here, considering that the high-precision map data lacks elevation information, the lane corner features of the lane lines at the current vehicle's location are not directly extracted from the high-precision map. Instead, lane line information at the current vehicle's location is obtained for comparison with lane lines extracted from the road image. Therefore, the lane line features extracted from the road image (i.e., the sequence of lane line sampling point coordinates) are projected onto the ground plane coordinate system using the aforementioned projection formula. This facilitates comparison with lane lines in the high-precision map data in the ground plane coordinate system. Specifically, the width and curvature values ​​between fitted lane lines are compared with those in the high-precision map data to generate an overall loss value. This value can then be used to characterize the overall error between the extracted lane lines and those in the high-precision map data. Here, the high-precision map data can be obtained by determining the lane of the road where the current vehicle is located using its current coordinates. Therefore, it is not necessary to equip the current vehicle with a highly accurate (e.g., centimeter-level) positioning device. Next, considering that the constructed ground-plane coordinate system is built from the pre-acquired camera height values ​​and camera coordinates, the accuracy of the ground-plane coordinate system cannot be guaranteed. Therefore, using the formula for generating the target transformation matrix described above, the overall loss value is used to adjust the transformation matrix of the camera coordinate system relative to the ground-plane coordinate system. Here, because the overall loss value is added, it can be used to eliminate the overall error between the fitted lane lines obtained after projecting the lane line sampling point coordinates to the ground-plane coordinate system and the lane lines in the high-precision map data. This improves the accuracy of the transformation matrix, and also improves the accuracy of the roll angle, pitch angle, and height values. Thus, the target transformation matrix and target height values ​​can be obtained.Therefore, by updating the aforementioned ground plane coordinate system using the target height value to obtain the target ground plane coordinate system, the accuracy of the target ground plane coordinate system can be improved. Then, by using the inverse of the target transformation matrix to project the lane line sampling points in the lane line sampling point coordinate sequence onto the target ground plane coordinate system, the accuracy of the target lane line sampling point coordinates can be improved. Furthermore, this can be used to improve the accuracy of the generated 3D lane lines.

[0064] Step 106: Perform lane line fitting on each target lane line sampling point in each target lane line sampling point coordinate group sequence to generate a three-dimensional lane line equation, thus obtaining a three-dimensional lane line equation set.

[0065] In some embodiments, the execution entity may perform lane line fitting on each target lane line sampling point in each target lane line sampling point coordinate group in the target lane line sampling point coordinate group sequence to generate a three-dimensional lane line equation, thereby obtaining a three-dimensional lane line equation group.

[0066] Optionally, the aforementioned executing entity may also send the aforementioned three-dimensional lane line equations to a display terminal for display.

[0067] The above-described embodiments of this disclosure have the following beneficial effects: The three-dimensional lane line generation method of some embodiments of this disclosure can generate three-dimensional lane lines even when lane line corner points extracted from road images are missing. Specifically, the reason why three-dimensional lane lines cannot be generated is that if lane line corner points cannot be extracted from the road image, it is difficult to solve the position and pose matrix of the camera relative to the ground. Based on this, the three-dimensional lane line generation method of some embodiments of this disclosure first extracts sampling points from the pre-acquired road image to obtain a sequence of lane line sampling point coordinates. Here, lane line sampling points can be coordinates with different features from lane line corner points, and both can be used to characterize the features of the lane line. Thus, the lane line sampling point coordinates obtained through sampling point extraction can be used to replace the lane line corner points that cannot be extracted for three-dimensional lane line generation. Second, based on a preset camera pose matrix, coordinate system transformation matrix, and pre-acquired current vehicle coordinate values, coordinate projection is performed on the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the above-described lane line sampling point coordinate group sequence to generate a projected lane line sampling point coordinate group, resulting in a sequence of projected lane line sampling point coordinate groups. Coordinate projection allows for the initial determination of the 3D coordinates corresponding to the extracted lane line corner points, i.e., the coordinates of the projected lane line sampling points. Then, curve fitting is performed on the coordinates of each projected lane line sampling point in each of the projected lane line sampling point coordinate groups to generate fitted lane lines, resulting in a fitted lane line sequence. Curve fitting can be used to fit 3D lane line equations in the vehicle coordinate system. Subsequently, based on the fitted lane line sequence and the map lane line information sequence from the preset high-precision map data that matches the current vehicle coordinate values, the coordinate system transformation matrix is ​​adjusted to generate the target transformation matrix. By introducing the map lane line information sequence from the high-precision map data, the coordinate system transformation matrix can be adjusted, thereby improving the accuracy of the target transformation matrix. Because the accuracy of the target transformation matrix is ​​improved, it can be used to characterize the pose matrix of the camera coordinate system relative to the ground plane coordinate system. Thus, a more accurate pose matrix is ​​obtained without relying on lane line corner points. Next, using the aforementioned target transformation matrix, coordinate projection is performed on the coordinates of each lane line sampling point in each lane line sampling point coordinate group sequence to generate the target lane line sampling point coordinate group, resulting in the target lane line sampling point coordinate group sequence. Here, provided the accuracy of the target transformation matrix is ​​ensured, further coordinate projection can improve the accuracy of the target lane line sampling point coordinates. Finally, lane line fitting is performed on each target lane line sampling point in each target lane line sampling point coordinate group sequence to generate a three-dimensional lane line equation, resulting in a three-dimensional lane line equation set.Therefore, it is possible to generate 3D lane lines without relying on lane line corner points, and to improve the accuracy of the generated 3D lane lines.

[0068] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a three-dimensional lane line generation device, which are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0069] like Figure 2 As shown, a 3D lane line generation device 200 in some embodiments includes: an extraction unit 201, a first coordinate projection unit 202, a first fitting unit 203, an adjustment processing unit 204, a second coordinate projection unit 205, and a second fitting unit 206. The extraction unit 201 is configured to extract sampling points from a pre-acquired road image to obtain a sequence of lane line sampling point coordinate groups; the first coordinate projection unit 202 is configured to perform coordinate projection on the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the above-mentioned lane line sampling point coordinate group sequence based on a preset camera pose matrix, a coordinate system transformation matrix, and pre-acquired current vehicle coordinate values ​​to generate a projected lane line sampling point coordinate group sequence; the first fitting unit 203 is configured to perform curve fitting on the coordinates of each projected lane line sampling point in each projected lane line sampling point coordinate group in the above-mentioned projected lane line sampling point coordinate group sequence to generate a fitted lane line, obtaining a fitted lane line sequence; the adjustment processing unit 204... The first unit is configured to adjust the coordinate system transformation matrix based on the fitted lane line sequence and the map lane line information sequence in the preset high-precision map data that matches the current vehicle coordinate values, so as to generate a target transformation matrix; the second coordinate projection unit 205 is configured to use the target transformation matrix to project the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the above lane line sampling point coordinate group sequence to generate a target lane line sampling point coordinate group, thereby obtaining a target lane line sampling point coordinate group sequence; the second fitting unit 206 is configured to perform lane line fitting on each target lane line sampling point in each target lane line sampling point in the above target lane line sampling point coordinate group sequence to generate a three-dimensional lane line equation, thereby obtaining a three-dimensional lane line equation group.

[0070] It is understandable that the units described in the device 200 are related to the reference. Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units contained therein, and will not be repeated here.

[0071] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0072] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0073] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0074] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0075] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0076] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0077] The aforementioned computer-readable medium may be included in the aforementioned device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: extract sampling points from a pre-acquired road image to obtain a sequence of lane line sampling point coordinates; based on a preset camera pose matrix, coordinate system transformation matrix, and pre-acquired current vehicle coordinate values, perform coordinate projection on the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the aforementioned sequence to generate a projected lane line sampling point coordinate group, obtaining a sequence of projected lane line sampling point coordinates; and perform curve fitting on the coordinates of each projected lane line sampling point in each projected lane line sampling point coordinate group in the aforementioned sequence to generate a simulated... The lane lines are combined to obtain a fitted lane line sequence. Based on the fitted lane line sequence and the map lane line information sequence that matches the current vehicle coordinates included in the preset high-precision map data, the coordinate system transformation matrix is ​​adjusted to generate a target transformation matrix. Using the target transformation matrix, the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the above lane line sampling point coordinate group sequence are projected to generate a target lane line sampling point coordinate group, resulting in a target lane line sampling point coordinate group sequence. Lane lines are fitted to each target lane line sampling point in each target lane line sampling point in the above target lane line sampling point coordinate group sequence to generate a three-dimensional lane line equation, resulting in a three-dimensional lane line equation set.

[0078] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0079] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0080] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an extraction unit, a first coordinate projection unit, a first fitting unit, an adjustment processing unit, a second coordinate projection unit, and a second fitting unit. The names of these units do not necessarily limit the specific unit; for example, the extraction unit may also be described as "a unit for extracting sampling points from a pre-acquired road image."

[0081] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0082] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for generating three-dimensional lane lines, comprising: Sampling points are extracted from the pre-acquired road image to obtain a sequence of lane line sampling point coordinates; Based on the preset camera pose matrix, coordinate system transformation matrix and pre-acquired current vehicle coordinate values, coordinate projection is performed on the coordinates of each lane line sampling point in each lane line sampling point coordinate group sequence to generate a projected lane line sampling point coordinate group, thus obtaining a projected lane line sampling point coordinate group sequence. Curve fitting is performed on the coordinates of each projected lane line sampling point in each projected lane line sampling point coordinate group sequence to generate a fitted lane line, thus obtaining a fitted lane line sequence. Based on the fitted lane line sequence and the map lane line information sequence that matches the current vehicle coordinate value included in the preset high-precision map data, the coordinate system transformation matrix is ​​adjusted to generate the target transformation matrix. Using the target transformation matrix, coordinate projection is performed on the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the lane line sampling point coordinate group sequence to generate the target lane line sampling point coordinate group, thus obtaining the target lane line sampling point coordinate group sequence. For each target lane line sampling point in each target lane line sampling point coordinate group sequence, lane line fitting is performed to generate a three-dimensional lane line equation, resulting in a three-dimensional lane line equation set. The step of adjusting the coordinate system transformation matrix based on the fitted lane line sequence and the map lane line information sequence in the preset high-precision map data that matches the current vehicle coordinate values ​​to generate a target transformation matrix includes: Based on the fitted lane line sequence and the map lane line information sequence that matches the current vehicle coordinate value included in the preset high-precision map data, a lane line width error value sequence and a lane line curvature error value sequence are generated. Based on the lane width error value sequence and the lane curvature error value sequence, the coordinate system transformation matrix is ​​adjusted to generate the target transformation matrix; The step of adjusting the coordinate system transformation matrix based on the lane width error value sequence and the lane curvature error value sequence to generate the target transformation matrix includes: Obtain the preset weight coefficients and the camera relative pose matrix; The coordinate system transformation matrix is ​​decomposed to generate the vehicle roll angle and vehicle pitch angle of the current vehicle. Based on the lane width error value sequence, the lane curvature error value sequence, the weighting coefficient, and the camera relative pose matrix, the vehicle roll angle, the vehicle pitch angle, and the camera height value are adjusted to generate a target transformation matrix and a target height value.

2. The method according to claim 1, wherein, The method further includes: The three-dimensional lane line equations are sent to the display terminal for display.

3. The method according to claim 1, wherein, Based on a preset camera pose matrix, coordinate system transformation matrix, and pre-acquired current vehicle coordinate values, coordinate projection is performed on the coordinates of each lane line sampling point in each lane line sampling point coordinate group sequence to generate a projected lane line sampling point coordinate group, resulting in a projected lane line sampling point coordinate group sequence, including: A ground plane coordinate system is constructed using the current vehicle coordinates and the pre-acquired camera height values; Based on the preset camera pose matrix, coordinate system transformation matrix, and pre-acquired current vehicle coordinate values, the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the lane line sampling point coordinate group sequence are projected onto the ground plane coordinate system to generate a projected lane line sampling point coordinate group, thus obtaining a projected lane line sampling point coordinate group sequence.

4. The method according to claim 1, wherein, Each map lane line information in the map lane line information sequence includes a map lane line; and The process of generating lane width error value sequences and lane curvature error value sequences based on the fitted lane line sequence and the map lane line information sequence that matches the current vehicle coordinates included in the preset high-precision map data includes: Determine the fitted lane width value between every two adjacent fitted lane lines in the fitted lane line sequence to obtain a fitted lane line width value sequence. Determine the map lane width value between each pair of adjacent map lane line information in the map lane line information sequence to obtain the map lane line width value sequence; The difference between each fitted lane width value and the corresponding map lane width value in the fitted lane width value sequence is determined as the lane width error value, thus obtaining the lane width error value sequence. Determine the curvature value of each fitted lane line in the fitted lane line sequence to obtain a sequence of fitted lane line curvature values. Determine the map lane curvature value of each map lane information included in the map lane information sequence to obtain the map lane curvature value sequence; The difference between each fitted lane curvature value and the corresponding map lane curvature value in the fitted lane width value sequence is determined as the lane curvature error value, thus obtaining the lane curvature error value sequence.

5. The method according to claim 3, wherein, The step of using the target transformation matrix to project the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the lane line sampling point coordinate group sequence to generate the target lane line sampling point coordinate group, resulting in the target lane line sampling point coordinate group sequence, includes: Based on the target height value, the ground plane coordinate system is updated to obtain the target ground plane coordinate system; Using the inverse of the target transformation matrix, the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the lane line sampling point coordinate group sequence are projected onto the target ground plane coordinate system to generate the target lane line sampling point coordinate group, thus obtaining the target lane line sampling point coordinate group sequence.

6. A three-dimensional lane line generation device, comprising: The extraction unit is configured to extract sampling points from the pre-acquired road image to obtain a sequence of lane line sampling point coordinates. The first coordinate projection unit is configured to project the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the lane line sampling point coordinate group sequence based on the preset camera pose matrix, coordinate system transformation matrix and pre-acquired current vehicle coordinate values ​​to generate a projected lane line sampling point coordinate group, thus obtaining a projected lane line sampling point coordinate group sequence. The first fitting unit is configured to perform curve fitting on the coordinates of each projected lane line sampling point in each projected lane line sampling point coordinate group in the sequence of projected lane line sampling point coordinate groups to generate a fitted lane line, thereby obtaining a fitted lane line sequence. The adjustment processing unit is configured to adjust the coordinate system transformation matrix based on the fitted lane line sequence and the map lane line information sequence in the preset high-precision map data that matches the current vehicle coordinate value, so as to generate a target transformation matrix. The second coordinate projection unit is configured to use the target transformation matrix to perform coordinate projection on the coordinates of each lane line sampling point in each lane line sampling point coordinate group in the lane line sampling point coordinate group sequence to generate a target lane line sampling point coordinate group, thereby obtaining a target lane line sampling point coordinate group sequence. The second fitting unit is configured to perform lane line fitting on each target lane line sampling point in each target lane line sampling point coordinate group in the target lane line sampling point coordinate group sequence to generate a three-dimensional lane line equation, thereby obtaining a three-dimensional lane line equation group.

7. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

8. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images