Lane line ground truth generation method and device, vehicle, and storage medium

Abstract

This application relates to the field of intelligent driving technology, and in particular to a method, apparatus, vehicle, and storage medium for generating lane line ground truth values. The method includes: acquiring a first set of three-dimensional lane lines; transforming the first set to a lidar coordinate system based on a first pose of a lidar sensor to obtain a second set, and projecting the second set onto a camera coordinate system to obtain a third set; calculating the camera pose with the smallest projection error based on the second and third sets; determining the second pose of the lidar sensor based on the camera pose; and generating the ground truth values ​​of two-dimensional lane lines based on the first pose, the second pose, and the second set. This solves the problems in related technologies such as over-reliance on sensor data synchronization and extrinsic parameter accuracy, and the difficulty and inefficiency of annotation due to sparse point clouds in single frames.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a method, apparatus, vehicle, and storage medium for generating lane line truth values. Background Technology

[0002] With the continuous development of autonomous driving technology, lane detection technology is also constantly improving. Lane detection using neural networks based on BEV (Bird's-eye-view Perception) perception has become very popular in recent years. Training of BEV-based neural networks relies on the three-dimensional true values ​​of objects in a two-dimensional image. Typically, point clouds are collected using LiDAR, and the two-dimensional lane lines are labeled manually.

[0003] However, existing 2D lane line annotation technologies are highly dependent on sensor data synchronization and the accuracy of extrinsic parameters between LiDAR and images; single-frame scan point cloud data may be sparse, making accurate lane line annotation difficult. Furthermore, methods for creating lane lines by annotating lane lines in single-frame LiDAR point clouds are relatively inefficient and time-consuming. Summary of the Invention

[0004] This application provides a method, apparatus, vehicle, and storage medium for generating lane line truth values ​​to solve problems in related technologies, such as over-reliance on sensor data synchronization and extrinsic parameter accuracy, and the difficulty and inefficiency of annotation due to sparse point clouds in single frames.

[0005] The first aspect of this application provides a method for generating true values ​​of lane lines, comprising the following steps: obtaining a first set of three-dimensional lane lines; transforming the first set to a lidar coordinate system based on a first pose of a lidar to obtain a second set, and projecting the second set onto a camera coordinate system to obtain a third set; calculating the camera pose with the smallest projection error based on the second set and the third set, determining the second pose of the lidar based on the camera pose, and generating true values ​​of two-dimensional lane lines based on the first pose, the second pose, and the second set.

[0006] Optionally, the step of projecting the second set onto the camera coordinate system to obtain the third set includes: obtaining the lidar extrinsic parameters, camera intrinsic parameters, and camera extrinsic parameters; determining the projection relationship based on the lidar extrinsic parameters, the camera intrinsic parameters, and the camera extrinsic parameters; and using the projection relationship to project the second set onto the camera coordinate system to obtain the third set.

[0007] Optionally, the step of calculating the camera pose with the minimum projection error based on the second set and the third set includes: constructing a point pair set of the second set and the third set; and calculating the camera pose with the minimum projection error of the camera in the radar coordinate system based on the point pair set.

[0008] Optionally, determining the second pose of the lidar based on the camera pose includes: acquiring lidar extrinsic parameters and camera extrinsic parameters; calculating the pose relationship between the camera and the lidar based on the lidar extrinsic parameters and the camera extrinsic parameters; and determining the second pose of the lidar based on the camera pose and the pose relationship.

[0009] Optionally, generating the ground truth of the two-dimensional lane line based on the first pose, the second pose, and the second set includes: calculating the difference between the first pose and the second pose; and optimizing the second set according to the difference to obtain the ground truth of the two-dimensional lane line.

[0010] Optionally, before obtaining the first set of three-dimensional lane lines, the method further includes: obtaining the vehicle pose and the laser point cloud frame of the LiDAR; determining the first pose of the LiDAR based on the vehicle pose and the laser point cloud frame; generating a multi-frame stitched point cloud based on the laser point cloud frame, the first pose, and / or a local map of the area where the vehicle is currently located; and performing three-dimensional lane line annotation on the multi-frame stitched point cloud to obtain the first set.

[0011] Optionally, generating a multi-frame stitched point cloud based on the laser point cloud frame, the first pose, and / or a local map of the area where the vehicle is currently located includes: identifying a local map of the area where the vehicle is currently located; if the local map is not identified, creating the local map and inserting the laser point cloud frame and the first pose into the local map to obtain the multi-frame stitched point cloud; if the local map is identified, globally optimizing the first pose based on the laser point cloud frame and the local map, and inserting the optimized first pose and the laser point cloud frame into the local map and the global map to obtain the multi-frame stitched point cloud.

[0012] A second aspect of this application provides a lane line truth value generation apparatus, comprising: an acquisition module for acquiring a first set of three-dimensional lane lines; a projection module for transforming the first set to a lidar coordinate system based on a first pose of a lidar to obtain a second set, and projecting the second set to a camera coordinate system to obtain a third set; a calculation module for calculating a camera pose with the minimum projection error based on the second set and the third set; and a generation module for determining a second pose of the lidar based on the camera pose, and generating two-dimensional lane line truth values ​​based on the first pose, the second pose, and the second set.

[0013] A third aspect of this application provides a vehicle, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the lane line truth generation method as described in the above embodiments.

[0014] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the lane line truth generation method as described in the above embodiments.

[0015] Therefore, this application has at least the following beneficial effects: This application's embodiments can annotate lane lines on multi-frame stitched point clouds, overcoming the shortcomings of sparse single-frame point clouds and low annotation efficiency. Simultaneously, it optimizes the pose change between laser point cloud frames, improving the correction of laser point cloud motion distortion and reducing the requirements for pose sensors. By optimizing the solution of the point pair set for camera pose, it accurately determines the camera's pose in the radar coordinate system, thereby reducing the requirements for time synchronization accuracy and extrinsic parameter calibration accuracy. Thus, it solves the technical problems in related technologies, such as over-reliance on sensor data synchronization and extrinsic parameter accuracy, and the difficulty and inefficiency of annotation due to sparse single-frame point clouds.

[0016] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0017] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is an example image of a single-frame laser point cloud in the existing technology; Figure 2 This is a flowchart of a lane line truth value generation method provided according to an embodiment of this application; Figure 3 This is a flowchart illustrating the truth value optimization of two-dimensional lane lines according to an embodiment of this application; Figure 4 This is a flowchart of laser point cloud stitching provided according to an embodiment of this application; Figure 5 This is an example diagram of lane line marking using laser point cloud technology according to an embodiment of this application; Figure 6 This is a flowchart illustrating the creation of a local map according to an embodiment of this application; Figure 7 This is an example diagram of a three-dimensional lane line projection provided according to an embodiment of this application; Figure 8 This is an example diagram of a two-dimensional lane line projection provided according to an embodiment of this application; Figure 9 This is a flowchart of a lane line truth value generation method according to an embodiment of this application; Figure 10 This is an example diagram of a lane line truth value generation apparatus according to an embodiment of this application; Figure 11 This is a structural schematic diagram of a vehicle according to an embodiment of this application. Detailed Implementation

[0018] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0019] Detecting lane lines using a neural network based on BEV perception is an advanced technology that relies on real-world data for training. During training, the neural network needs to be inputted with 3D ground truth information from a 2D image, including the object's pose and contour extent. This ground truth is typically obtained by acquiring point cloud data using LiDAR and adding manual annotations. Similar methods can also be used for 2D lane line detection.

[0020] However, the related technologies have high requirements for sensor data synchronization and the accuracy of lidar and image extrinsic parameters, meaning that the lidar point cloud and the image need to be acquired at the same time. With high accuracy, a near one-to-one correspondence can be achieved when the 3D point cloud is projected onto the 2D image. But if the accuracy is low, this one-to-one correspondence cannot be achieved, thus requiring manual adjustment.

[0021] For ordinary vehicle-mounted 3D LiDAR, the point cloud from a single frame scan is relatively sparse, such as Figure 1 As shown, it is difficult to accurately identify the contours of lane lines. To improve recognition accuracy, high-precision pose sensors are usually needed to compensate for distortions in the laser point cloud caused by motion. However, identifying lane lines by simply labeling a single frame of laser point cloud is not only inefficient but also time-consuming.

[0022] The following description, with reference to the accompanying drawings, outlines a lane line ground truth generation method, apparatus, vehicle, and storage medium according to embodiments of this application. Addressing the issues mentioned in the background art, such as the excessive reliance on image extrinsic parameter accuracy leading to low lane line annotation accuracy, this application provides a lane line ground truth generation method. This method allows lane line annotation on multi-frame stitched point clouds, overcoming the drawbacks of sparse single-frame point clouds and low annotation efficiency. It also optimizes the pose change between laser point cloud frames, improving the correction of laser point cloud motion distortion and reducing the requirements for pose sensors. By optimizing the solution of the point pair set for camera pose, the camera pose in the radar coordinate system is accurately determined, thereby reducing the requirements for time synchronization accuracy and extrinsic parameter calibration accuracy. Thus, it solves the problems of excessive reliance on sensor data synchronization and extrinsic parameter accuracy, and the difficulty and inefficiency of annotation due to sparse single-frame point clouds in related technologies.

[0023] Specifically, Figure 2 This is a flowchart illustrating a method for generating true values ​​for lane lines provided in an embodiment of this application.

[0024] like Figure 2 As shown, the method for generating true values ​​for lane lines includes the following steps: In step S101, the first set of three-dimensional lane lines is obtained.

[0025] The first set can include information such as the geometry, location, and height of the three-dimensional lane lines. After clustering, a representative feature set is formed.

[0026] It is understood that the embodiments of this application can obtain a first set of three-dimensional lane lines, providing accurate lane line information. The first set can be determined according to actual circumstances; for example, the first set can be a set of starting points of the three-dimensional lane line dashed lines, and the set of starting points of the dashed lines can be Dash3d.

[0027] It should be noted that when the first set is the set of starting points of the dashed lane lines in 3D, and the Dash3d is sufficiently small, the set of solid line points is obtained. The set of solid line points can be Line3d. The set of solid line points can be used to determine the driving lane of the vehicle and constrain the lateral position and driving direction of the vehicle according to the lane line information to ensure the vehicle drives safely on the road.

[0028] For example, such as Figure 3 As shown, the set of starting points of the dashed lines in the stitched 3D lane lines, which are manually or automatically labeled, is obtained (Dash3d). The local coordinate system is stitched together over a period of time. The local coordinate system can be a local geodetic coordinate system. For example, the pose at the start of the acquisition can be (0, 0, 0).

[0029] It should be noted that the set of solid line points in the 3D lane line is obtained as Line3d.

[0030] In this embodiment of the application, before obtaining the first set of three-dimensional lane lines, the method further includes: obtaining the vehicle pose and the laser point cloud frame of the lidar; determining the first pose of the lidar based on the vehicle pose and the laser point cloud frame; generating a multi-frame stitched point cloud based on the laser point cloud frame, the first pose and / or the local map of the area where the vehicle is currently located; and performing three-dimensional lane line annotation on the multi-frame stitched point cloud to obtain the first set.

[0031] The vehicle pose can include its position and orientation in three-dimensional space, and the first pose can be an estimate of the position and orientation of the LiDAR relative to the vehicle.

[0032] It is understandable that, such as Figure 4 As shown, this embodiment of the application can determine the first pose of the lidar by the vehicle's pose and the lidar point cloud frame, and then stitch together and annotate the point cloud data of multiple frames, overcoming the shortcomings of sparse point cloud in a single frame and low annotation efficiency in a single frame, optimizing the pose change between lidar point cloud frames, improving the distortion correction of the lidar point cloud, and reducing the dependence on the pose sensor. Figure 5 As shown, the starting point of the lane line is marked on the laser point cloud, which can obtain the real lane line coordinates in the three-dimensional world and use them as the true value of the two-dimensional lane line.

[0033] It should be noted that data is collected through a combination of LiDAR and RTK (Real-time dynamic carrier phase differential technology), GNSS (Global Navigation Satellite System), IMU (Inertial Measurement Unit), or wheel speedometer modules to construct a local map. The data source can provide information such as the vehicle's position, attitude, and speed, as well as detailed point cloud data of the vehicle's surrounding environment.

[0034] In this embodiment of the application, generating a multi-frame stitched point cloud based on the laser point cloud frame, the first pose, and / or the local map of the area where the vehicle is currently located includes: identifying the local map of the area where the vehicle is currently located; if no local map is identified, creating a local map and inserting the laser point cloud frame and the first pose into the local map to obtain a multi-frame stitched point cloud; if a local map is identified, globally optimizing the first pose based on the laser point cloud frame and the local map, and inserting the optimized first pose and the laser point cloud frame into the local map and the global map to obtain a multi-frame stitched point cloud.

[0035] It is understandable that, such as Figure 4As shown, this embodiment of the application can identify a local map of the area where the vehicle is currently located by sensing the environment around the vehicle through sensors. If no local map is identified, a local map is created, such as... Figure 6 As shown, creating a local map involves performing laser point cloud motion compensation on LiDAR data and RTK pose data to correct the distance of the point cloud. Then, feature extraction is performed to infer vehicle pose changes, and optimization is applied to obtain a collection of local point cloud maps, enabling effective environmental perception. When a local map is identified, the LiDAR point cloud frame and the local map can be used to globally optimize the first pose, inserting it along with subsequent LiDAR point cloud frames into both the local and global maps. Global optimization reduces reliance on pose sensor data, improving robustness and stability.

[0036] In step S102, the first set is transformed into the lidar coordinate system based on the first pose of the lidar to obtain the second set, and the second set is projected into the camera coordinate system to obtain the third set.

[0037] The second set can be Dash3d_new, etc., and the third set can be Dash3d_camera, etc.

[0038] It is understood that the embodiments of this application can transform the first set to the second set and project the second set onto the camera coordinate system to obtain the third set, so as to better utilize the environmental information acquired by the camera, realize data fusion between the lidar and the camera, and thus obtain a more comprehensive and accurate environmental perception result.

[0039] For example, such as Figure 3 As shown, when the first set Dash3d and Line3d are transformed into the LiDAR coordinate system to obtain the second set Dash3d_new and Line3d_new, the transformation method is as follows: Dash3d_new=LidarPose.inverse()*Dash3d, Line3d_new=LidarPose.inverse()*Line3d, Here, Dash3d_new represents the second set, Dash3d represents the first set, and LidarPose.inverse() represents the transformation of the lidar coordinate system.

[0040] like Figure 7 As shown, the short lines represent the first set. The pixel coordinates of the first set projected onto the camera image yield the second set. The longer the line connecting the second set to the starting point of the lane line marked on the image itself, the greater the error.

[0041] In this embodiment of the application, projecting the second set onto the camera coordinate system to obtain the third set includes: obtaining the lidar extrinsic parameters, camera intrinsic parameters, and camera extrinsic parameters; determining the projection relationship based on the lidar extrinsic parameters, camera intrinsic parameters, and camera extrinsic parameters; and using the projection relationship to project the second set onto the camera coordinate system to obtain the third set.

[0042] Among them, the lidar extrinsic parameters can be the transformation relationship between the lidar coordinate system and the world coordinate system, the camera intrinsic parameters can be the internal parameters of the camera, including focal length, principal point coordinates, etc., and the camera extrinsic parameters can be the transformation relationship of the camera relative to the world coordinate system. The camera extrinsic parameters can be calibrated as lidar2camera, etc.

[0043] It is understood that the embodiments of this application can determine the projection relationship based on the lidar extrinsic parameters, camera intrinsic parameters, and camera extrinsic parameters. Specifically, the point cloud data in the lidar coordinate system is first converted to the world coordinate system, and then the point cloud data in the world coordinate system is projected onto the camera coordinate system to obtain the third set.

[0044] For example, such as Figure 3 As shown, based on the camera's intrinsic and extrinsic parameters, the second set Dash3d_new and Line3d_new in the laser coordinate system are projected onto the camera coordinate system to obtain the third set Dash3d_camera and Line3d_camera.

[0045] In step S103, the camera pose with the smallest projection error is calculated based on the second set and the third set, the second pose of the lidar is determined based on the camera pose, and the true value of the two-dimensional lane line is generated based on the first pose, the second pose and the second set.

[0046] It is understood that the embodiments of this application can calculate the minimum projection error by using the second set and the third set. By calculating the camera pose with the minimum projection error, the accuracy and robustness of environmental perception can be improved. The second pose of the LiDAR can be determined based on the camera pose, and the true value of the two-dimensional lane line can be generated based on the first pose, the second pose, and the second set, which can reflect the actual position of the lane line.

[0047] In this embodiment of the application, calculating the camera pose with the minimum projection error based on the second set and the third set includes: constructing a point pair set of the second set and the third set; and calculating the camera pose with the minimum projection error in the radar coordinate system based on the point pair set.

[0048] It is understood that, in the embodiments of this application, each point in the first set is paired with each point in the third set to obtain a point-pair set. The projection error of each point pair in the point-pair set is compared to determine the camera pose, thereby realizing camera pose estimation based on point-pair sets, accurately determining the position and attitude of the camera in the radar coordinate system, and thus minimizing the projection error.

[0049] It should be noted that the embodiments of this application can also obtain a set of two-dimensional lane lines. The set of two-dimensional lane lines can be a set of starting points of two-dimensional lane line dashed lines, such as Dash2d. By comparing the two-dimensional lane lines with the third set, a second set of point pairs is obtained, thereby obtaining the basic correspondence between the two-dimensional lane lines and the three-dimensional lane line coordinates.

[0050] Meanwhile, in the embodiments of this application, the point pair set and the second point pair set can be solved by the PnP algorithm (Perspective-n-Point) to accurately determine the pose of the camera in the radar coordinate system, thereby reducing the requirements for time synchronization accuracy and extrinsic parameter calibration accuracy.

[0051] For example, such as Figure 3 As shown, the first set Dash3d corresponding to the third set Dash3d_camera is found, and a point-to-point set is constructed.

[0052] It should be noted that the process involves reading the set of dashed line starting points (Dash2d) and the set of solid line points (Line2d) from the 2D lane lines, as well as the pixel coordinates from the image. The closest points in Dash2d and the third set (Dash3d_camera) are then matched based on pixel distance to find the second set of point pairs, establishing a basic correspondence between the 2D lane lines and the 3D lane line coordinates.

[0053] In this embodiment of the application, determining the second pose of the LiDAR based on the camera pose includes: acquiring the LiDAR extrinsic parameters and the camera extrinsic parameters; calculating the pose relationship between the camera and the LiDAR based on the LiDAR extrinsic parameters and the camera extrinsic parameters; and determining the second pose of the LiDAR based on the camera pose and the pose relationship.

[0054] It is understood that the embodiments of this application can calculate the pose relationship between the camera and the lidar by obtaining the extrinsic parameters of the lidar and the camera, thereby determining the second pose of the lidar in the camera coordinate system, which helps to align and fuse the lidar data with the camera data.

[0055] For example, lidar and optical sensors are two different types of sensors. LiDAR acquires 3D point cloud data by emitting a laser beam and measuring the time it takes for it to reflect back, while optical sensors acquire 2D image data by capturing images. Using an object or scene with known position and orientation as a reference, the M_Lidar2Camera matrix can be optimized by measuring the observations of the same object by both lidar and optical sensors. The process of transforming the pose information of the lidar from the optical sensor coordinate system to the world coordinate system is calculated as follows: LidarPose=CamerPose*M_Lidar2Camera, Among them, CamerPose can be the pose information, and LidarPose can be the pose of the LiDAR.

[0056] In this embodiment of the application, generating the true value of a two-dimensional lane line based on a first pose, a second pose, and a second set includes: calculating the difference between the first pose and the second pose; and optimizing the second set based on the difference to obtain the true value of the two-dimensional lane line.

[0057] It is understood that the embodiments of this application can calculate the difference between the first pose and the second pose, optimize the second set based on the difference, and obtain the true value of the real two-dimensional lane line by correcting the second set, so as to ensure that the optimization result can be as close as possible to the position and shape of the real two-dimensional lane line.

[0058] For example, such as Figure 3 As shown, the difference between the first pose and the second pose is optimized to correct the second set Dash3d_new, thus obtaining Dash3d_real.

[0059] like Figure 8 As shown, the line connecting the optimized projection point and the starting point of the lane line marked on the 2D image is significantly shorter, reducing projection error and making the projection point closer to the actual starting point of the lane line.

[0060] The lane line ground truth generation method proposed in this application can annotate lane lines on multi-frame stitched point clouds, overcoming the shortcomings of sparse single-frame point clouds and low annotation efficiency. It also optimizes the pose change between laser point cloud frames, improving the correction of laser point cloud motion distortion and reducing the requirements for pose sensors. By optimizing the solution of the camera pose of the point pair set, the camera pose in the radar coordinate system is accurately determined, thereby reducing the requirements for time synchronization accuracy and extrinsic parameter calibration accuracy. Thus, it solves the problems of over-reliance on sensor data synchronization and extrinsic parameter accuracy, and the difficulty and inefficiency of annotation caused by sparse single-frame point clouds in related technologies.

[0061] The lane line truth value generation method of this application is illustrated below through a specific embodiment, such as... Figure 9 As shown, the method for generating true values ​​for lane lines includes the following steps: S1. Obtain the set of starting points of the dashed lines in the stitched 3D lane lines, either manually or automatically labeled, such as {(3d_dash_start0,3d_dash_end0),(3d_dash_start1,3d_dash_end1),...,(3d_dash_start_m,3d_dash_end_m)}; stitch together a local coordinate system over a period of time, where the local coordinate system can be a local geodetic coordinate system; It should be noted that the set of solid line points Line3d in the 3D lane line is obtained, for example, {3d_line_0, 3d_line_1, ..., 3d_line_n}; Read the set of starting points of the dashed lines in the 2D lane lines, Dash2d, and the pixel coordinates in the image, such as {(2d_dash_start0,2d_dash_end0),(2d_dash_start1,2d_dash_end1),...,(2d_dash_start_t,2d_dash_end_t)}; Read the set of solid line points Line2d in the 2D lane line, for example, {2d_line_0, 2d_line_1, ..., 2d_line_l}.

[0062] S2. Obtain the first pose after optimization at the current data frame time.

[0063] S3. Based on the pose, convert the labeled 3D lane lines Dash3d and Line3d to Dash3d_new and Line3d_new in the LiDAR coordinate system. The conversion method is as follows: Dash3d_new=LidarPose.inverse()*Dash3d, Line3d_new=LidarPose.inverse()*Line3d, Here, Dash3d_new can be the second set, Dash3d can be the first set, and LidarPose.inverse() can be used to transform the coordinate system of the LiDAR.

[0064] S4. Based on the camera's intrinsic and extrinsic parameters, project the point sets Dash3d_new and Line3d_new in the laser coordinate system onto the Camera coordinate systems Dash3d_camera and Line3d_camera.

[0065] S5. Find the corresponding Dash3d based on Dash3d_camera and construct a point-to-point set, such as {(Point_Dash2d_0,Point_Dash3d_0),(Point_Dash2d_1,Point_Dash3d_1),...,(Point_Dash2d_k,Point_Dash3d_k)}; It should be noted that, based on the pixel distance between Dash2d and Dash3d_camera, the nearest point is matched to find the second set of point pairs, establishing a basic correspondence with Dash3d. The second set of point pairs can be {(Point_Dash2d_0,Point_Dash3d_camera_0),(Point_Dash2d_1,Point_Dash3d_camera_1),...,(Point_Dash2d_k,Point_Dash3d_camera_k)}.

[0066] S6. Solve the point set using solvePnP to calculate the pose with the minimum projection error in the Dash3d local geodetic coordinate system.

[0067] S7. Calculate the second pose based on the extrinsic parameters of the lidar and the camera.

[0068] S8. Optimize the difference between the first pose and the second pose to correct Dash3d_new and obtain Dash3d_real.

[0069] Next, the lane line truth value generation apparatus according to an embodiment of this application is described with reference to the accompanying drawings.

[0070] Figure 10 This is a block diagram of a lane line truth value generation device according to an embodiment of this application.

[0071] like Figure 10 As shown, the lane line truth value generation device 110 includes: an acquisition module 100, a projection module 200, a calculation module 300, and a generation module 400.

[0072] The acquisition module 100 is used to acquire a first set of three-dimensional lane lines; the projection module 200 is used to transform the first set to the lidar coordinate system based on the first pose of the lidar to obtain a second set, and project the second set to the camera coordinate system to obtain a third set; the calculation module 300 is used to calculate the camera pose with the minimum projection error based on the second set and the third set; and the generation module 400 is used to determine the second pose of the lidar based on the camera pose, and generate the true value of the two-dimensional lane lines based on the first pose, the second pose and the second set.

[0073] It should be noted that the foregoing explanation of the lane line truth value generation method embodiment also applies to the lane line truth value generation device of this embodiment, and will not be repeated here.

[0074] The lane line truth generation device proposed in this application can annotate lane lines on multi-frame stitched point clouds, overcoming the shortcomings of sparse single-frame point clouds and low annotation efficiency. It also optimizes the pose change between laser point cloud frames, improving the correction of laser point cloud motion distortion and reducing the requirements for pose sensors. By optimizing the solution of the camera pose in the point pair set, the camera pose in the radar coordinate system is accurately determined, thereby reducing the requirements for time synchronization accuracy and extrinsic parameter calibration accuracy. Thus, it solves the problems of over-reliance on sensor data synchronization and extrinsic parameter accuracy, and the difficulty and inefficiency of annotation caused by sparse single-frame point clouds in related technologies.

[0075] Figure 11 A schematic diagram of the structure of a vehicle provided in an embodiment of this application. The vehicle may include: The memory 1001, the processor 1002, and the computer program stored on the memory 1001 and capable of running on the processor 1002.

[0076] When the processor 1002 executes the program, it implements the lane line truth value generation method provided in the above embodiments.

[0077] Furthermore, the vehicle also includes: Communication interface 1003 is used for communication between memory 1001 and processor 1002.

[0078] The memory 1001 is used to store computer programs that can run on the processor 1002.

[0079] The memory 1001 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0080] If the memory 1001, processor 1002, and communication interface 1003 are implemented independently, then the communication interface 1003, memory 1001, and processor 1002 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0081] Optionally, in a specific implementation, if the memory 1001, processor 1002, and communication interface 1003 are integrated on a single chip, then the memory 1001, processor 1002, and communication interface 1003 can communicate with each other through an internal interface.

[0082] The processor 1002 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0083] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the lane line truth value generation method described above.

[0084] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0085] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0086] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0087] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0088] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0089] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A lane line ground truth generation method, characterized by, Includes the following steps: Obtain the first set of 3D lane lines; Based on the first pose of the lidar, the first set is transformed into the lidar coordinate system to obtain the second set, and the second set is projected into the camera coordinate system to obtain the third set; The camera pose with the smallest projection error is calculated based on the second set and the third set. The second pose of the lidar is determined based on the camera pose. The true value of the two-dimensional lane line is generated based on the first pose, the second pose and the second set. The step of calculating the camera pose with the minimum projection error based on the second set and the third set includes: Construct a pairwise set between the second set and the third set; Calculate the camera pose with the minimum projection error in the radar coordinate system based on the point pair set; The generation of the ground truth value for the two-dimensional lane line based on the first pose, the second pose, and the second set includes: Calculate the difference between the first pose and the second pose; The true value of the two-dimensional lane line is obtained by optimizing the second set based on the difference.

2. The lane line truth value generation method according to claim 1, characterized in that, The step of projecting the second set onto the camera coordinate system to obtain the third set includes: Acquire lidar extrinsic parameters, camera intrinsic parameters, and camera extrinsic parameters; The projection relationship is determined based on the lidar extrinsic parameters, the camera intrinsic parameters, and the camera extrinsic parameters; The second set is projected onto the camera coordinate system using the projection relationship to obtain the third set.

3. The lane line truth generation method according to claim 1, characterized in that, Determining the second pose of the lidar based on the camera pose includes: Acquire the extrinsic parameters of the lidar and the camera; The pose relationship between the camera and the lidar is calculated based on the lidar extrinsic parameters and the camera extrinsic parameters; The second pose of the lidar is determined based on the camera pose and the pose relationship.

4. The lane line truth value generation method according to claim 1, characterized in that, Before obtaining the first set of 3D lane lines, the following is also included: Obtain the vehicle pose and the laser point cloud frame of the lidar; The first pose of the lidar is determined based on the vehicle pose and the laser point cloud frame. A multi-frame stitched point cloud is generated based on the laser point cloud frame, the first pose, and / or the local map of the area where the vehicle is currently located. The multi-frame stitched point cloud is then annotated with three-dimensional lane lines to obtain the first set.

5. The lane line truth value generation method according to claim 4, characterized in that, The step of generating a multi-frame stitched point cloud based on the laser point cloud frame, the first pose, and / or the local map of the area where the vehicle is currently located includes: Identify a local map of the area where the vehicle is currently located; If the local map is not identified, the local map is created, and the laser point cloud frame and the first pose are inserted into the local map to obtain the multi-frame stitched point cloud; If the local map is identified, the first pose is globally optimized based on the laser point cloud frame and the local map, and the optimized first pose and the laser point cloud frame are inserted into the local map and the global map to obtain the multi-frame stitched point cloud.

6. A lane line truth value generation device, characterized in that, The device includes: The acquisition module is used to acquire the first set of three-dimensional lane lines; The projection module is used to transform the first set to the lidar coordinate system based on the first pose of the lidar to obtain the second set, and project the second set to the camera coordinate system to obtain the third set; The calculation module is used to calculate the camera pose with the minimum projection error based on the second set and the third set; The generation module is used to determine the second pose of the lidar based on the camera pose, and generate the ground truth of the two-dimensional lane line based on the first pose, the second pose and the second set; The step of calculating the camera pose with the minimum projection error based on the second set and the third set includes: Construct a pairwise set between the second set and the third set; Calculate the camera pose with the minimum projection error in the radar coordinate system based on the point pair set; The generation of the ground truth value for the two-dimensional lane line based on the first pose, the second pose, and the second set includes: Calculate the difference between the first pose and the second pose; The true value of the two-dimensional lane line is obtained by optimizing the second set based on the difference.

7. A vehicle, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the lane line truth generation method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the lane line truth generation method as described in any one of claims 1-5.

Images