Method and device for marking lane line and electronic equipment

Abstract

The invention provides a method and device for marking a lane line and electronic equipment, and the method comprises the steps: obtaining a splicing point cloud on a vehicle driving road, and enabling the splicing point cloud to be obtained through the splicing of a plurality of frames of point clouds collected when a vehicle drives on the vehicle driving road; the spliced point cloud is subjected to spatial transformation, a target point cloud image is obtained, and the target point cloud image is a projection image of the spliced point cloud on the ground; determining a first two-dimensional coordinate of a lane line in the target point cloud image; and determining a first three-dimensional coordinate of the lane line according to the first two-dimensional coordinate of the lane line and the splicing point cloud. According to the method, the efficiency of marking the lane line can be improved, and the cost of marking the lane line is reduced.

Description

Technical Field

[0001] This application relates to the field of vehicles, and more specifically, to a method, apparatus, and electronic device for marking lane lines in the field of vehicles. Background Technology

[0002] In the automotive industry, with the continuous updates of internet technology, vehicle functions are becoming increasingly sophisticated. For example, vehicles now possess autonomous driving capabilities.

[0003] In the process of achieving autonomous driving, vehicles need to be able to simultaneously meet the requirements of various driving scenarios such as automatic cruise, overtaking, parking, and deceleration. The decision-making and judgment of the aforementioned vehicles in various application scenarios are inseparable from the detection of lane lines, and an important prerequisite for lane line detection is the lane line marking process.

[0004] In related technologies, when marking lane lines, technicians usually manually mark the three-dimensional coordinates of the lane lines by referring to point cloud data.

[0005] However, the methods for marking lane lines in related technologies are inefficient and costly. Summary of the Invention

[0006] This application provides a method, apparatus, and electronic device for marking lane lines, which can improve the efficiency of lane line marking and reduce the cost of lane line marking.

[0007] In a first aspect, a method for marking lane lines is provided, the method comprising: acquiring a stitched point cloud on a road in which a vehicle travels, the stitched point cloud being obtained by stitching together multiple frames of point clouds collected when the vehicle travels on the road; performing a spatial transformation on the stitched point cloud to obtain a target point cloud image, the target point cloud image being a projection image of the stitched point cloud on the ground; determining the first two-dimensional coordinates of the lane lines in the target point cloud image; and determining the first three-dimensional coordinates of the lane lines based on the first two-dimensional coordinates of the lane lines and the stitched point cloud.

[0008] The aforementioned technical solution proposes a method for labeling lane lines. Specifically, it first stitches together multiple frames of point clouds along the road where the vehicle is traveling to obtain a stitched point cloud. This stitched point cloud is then projected to convert into a target point cloud image, i.e., a top-down view of the stitched point cloud parallel to the ground, achieving dimensionality reduction and decreasing the workload for subsequent labeling. The two-dimensional coordinates of the lane lines are obtained from the target point cloud image. This process simplifies lane line labeling to two-dimensional labeling, which is more efficient and less costly than directly labeling three-dimensional coordinates. Based on the first two-dimensional coordinates of the lane lines and the stitched point cloud, the first three-dimensional coordinates of the lane lines are determined. This means that the stitched point cloud automatically expands the already labeled two-dimensional coordinates into three-dimensional coordinates, significantly improving the efficiency of lane line labeling. In other words, the technical solution provided in this application essentially obtains the three-dimensional coordinates of the lane lines indirectly through their two-dimensional coordinates, thereby completing the three-dimensional labeling of the lane lines. Compared to directly labeling the three-dimensional coordinates of the lane lines, the method in this application can improve the efficiency and reduce the cost of lane line labeling.

[0009] In conjunction with the first aspect, in some possible implementations, determining the first two-dimensional coordinates of the lane line in the target point cloud image includes: annotating the target point cloud image using a preset annotation tool to obtain the first two-dimensional coordinates of the lane line.

[0010] In the above technical solution, when annotating lane lines in the target point cloud image, the main method used is to use a preset annotation tool. On the one hand, this can reduce the development cost in the lane line annotation process, and on the other hand, the lane line annotation speed can be improved by annotating lane lines in a two-dimensional way.

[0011] In combination with the first aspect and the above implementation methods, in some possible implementation methods, the first two-dimensional coordinates are the first pixel coordinates of the lane line in the target point cloud image, and determining the first three-dimensional coordinates of the lane line based on the first two-dimensional coordinates of the lane line and the stitched point cloud includes: determining the image coordinates of the lane line in the target point cloud image based on the first coordinate transformation relationship and the first pixel coordinates of the lane line; and determining the first three-dimensional coordinates of the lane line in the stitched point cloud coordinate system based on the image coordinates of the lane line and the stitched point cloud.

[0012] The above technical solution proposes a method for determining the first three-dimensional coordinates of lane lines. When labeling lane lines in a target point cloud image, the pixel coordinates (i.e., the first pixel coordinates) of the lane lines are generally labeled. However, the first pixel coordinates are not coordinates on the physical imaging plane. Therefore, it is first necessary to convert the first pixel coordinates into coordinates on the physical imaging plane, i.e., the image coordinates of the lane lines in the target point cloud image. Further, based on this, and combined with the stitched point cloud, the first three-dimensional coordinates of the lane lines in the stitched point cloud coordinate system can be obtained. Through the above process, the three-dimensional lane line coordinates can be obtained from the two-dimensional coordinates of the lane lines, thus transforming the lane line labeling problem from two-dimensional labeling to three-dimensional labeling, completing the process of labeling the three-dimensional coordinates of the lane lines.

[0013] In combination with the first aspect and the above implementation methods, in some possible implementation methods, determining the first three-dimensional coordinates of the lane line in the stitched point cloud coordinate system based on the image coordinates of the lane line and the stitched point cloud includes: for any point on the lane line, determining the height of the point in the stitched point cloud based on the image coordinates of the point; and determining the first three-dimensional coordinates of the lane line based on the image coordinates of the point and the height of the point in the stitched point cloud.

[0014] The above technical solution proposes a method to obtain the first three-dimensional coordinates of lane lines based on the image coordinates of lane lines in a target point cloud image and the stitched point cloud. Since the target point cloud image is a top view of the stitched point cloud, this application can determine the position of each point in the stitched point cloud coordinate system based on the image coordinates of each point of the lane line in the target point cloud image. Furthermore, based on the three-dimensional coordinates of each point in the stitched point cloud coordinate system, the first three-dimensional coordinates of the lane line in the stitched point cloud coordinate system are obtained. Through this process, in the process of annotating lane lines, the three-dimensional coordinates of the lane lines can be obtained by combining the two-dimensional coordinates of the lane lines with the height of each point cloud, thus realizing the conversion from two-dimensional annotation to three-dimensional annotation, thereby reducing the difficulty of lane line annotation and making the lane line annotation process simpler and more efficient.

[0015] In combination with the first aspect and the above implementation methods, in some possible implementation methods, after determining the first three-dimensional coordinates of the lane line in the stitched point cloud coordinate system based on the image coordinates and the stitched point cloud, the method further includes: determining the second three-dimensional coordinates of the lane line in the original point cloud coordinate system corresponding to the multi-frame point cloud based on the second coordinate transformation relationship and the first three-dimensional coordinates of the lane line; and determining the second two-dimensional coordinates of the lane line in the camera image plane of the vehicle based on the third coordinate transformation relationship and the second three-dimensional coordinates of the lane line.

[0016] In the above technical solutions, in addition to the point cloud acquisition equipment, the vehicle typically also includes a camera. Therefore, in this application, the obtained first three-dimensional coordinates can also be projected onto the camera image plane.

[0017] The first three-dimensional coordinates are the three-dimensional coordinates in the stitched point cloud coordinate system. Since the stitched point cloud is composed of multiple different point clouds from a LiDAR scanner, it is not the original multi-frame point cloud. Therefore, after obtaining the three-dimensional coordinates in the stitched point cloud coordinate system, this application needs to restore the first three-dimensional coordinates one by one, that is, transform each three-dimensional coordinate in the stitched point cloud to the original point cloud coordinate system to obtain the original lane line's second three-dimensional coordinates. Further, based on the second three-dimensional coordinates, each frame is projected onto the camera image plane to obtain the second two-dimensional coordinates in the camera image plane.

[0018] If there are a large number of vehicle cameras, the above process can project a single point cloud image onto multiple camera image planes at once, thereby obtaining the two-dimensional coordinates of lane lines in multiple camera image planes simultaneously and rapidly improving the efficiency of lane line labeling.

[0019] In combination with the first aspect and the above implementation methods, in some possible implementation methods, the second two-dimensional coordinate is the second pixel coordinate of the lane line in the camera image plane. Determining the second two-dimensional coordinate of the lane line in the camera image plane of the vehicle according to the third coordinate transformation relationship and the second three-dimensional coordinate of the lane line includes: determining the third three-dimensional coordinate of the lane line in the camera coordinate system according to the fourth coordinate transformation relationship and the second three-dimensional coordinate of the lane line; and determining the second pixel coordinate of the lane line according to the intrinsic parameter matrix of the camera and the fifth coordinate transformation relationship.

[0020] The above technical solution specifically proposes a process for determining the second two-dimensional coordinates of the lane line in the camera image plane. Here, the second two-dimensional coordinates are the second pixel coordinates in the camera image plane. First, the second three-dimensional coordinates of the lane line are converted into the third three-dimensional coordinates of the lane line in the camera coordinate system. This process transforms the lane line coordinates from the original point cloud coordinate system to the camera coordinate system, thereby achieving projection of the lane line onto the camera plane based on point cloud data. Furthermore, in camera imaging, the image captured by the camera is ultimately on the imaging plane. Therefore, it is necessary to further combine the coordinate transformation relationship between camera coordinates and image coordinates to obtain the second pixel coordinates of the lane line in the camera image plane, thus completing the projection of the lane line onto the camera plane.

[0021] In combination with the first aspect and the above implementation methods, in some possible implementation methods, the spatial transformation of the stitched point cloud to obtain the target point cloud image includes: segmenting the stitched point cloud into at least one voxel; projecting each voxel in the at least one voxel as a point to obtain the target point cloud image.

[0022] In the above technical solution, voxelization is specifically used in the process of converting the stitched point cloud into a target point cloud image. First, the stitched point cloud is divided into at least one voxel. Then, the target point cloud image is obtained by projecting each voxel onto the plane containing the target point cloud image.

[0023] In combination with the first aspect and the above implementation methods, in some possible implementation methods, the acquisition of the stitched point cloud on the vehicle's driving road includes: based on a point cloud registration algorithm, performing rotation processing and / or translation processing on the multi-frame point cloud to obtain the stitched point cloud.

[0024] In the above technical solution, the stitched point cloud is obtained by stitching together multiple frames of point clouds. Specifically, in the stitching process of this application, based on the principle of point cloud stitching, the multiple frames of point clouds are rotated and translated to unify them into the same coordinate system, thus obtaining the stitched point cloud, which provides a basis for subsequent lane line annotation based on the multiple frames of point clouds.

[0025] In summary, this application proposes a method for labeling lane lines. First, multiple frames of point clouds along the road are stitched together to obtain a stitched point cloud. This stitched point cloud is then projected to convert it into a target point cloud image, i.e., a top-down view of the stitched point cloud parallel to the ground, achieving dimensionality reduction and decreasing the workload for subsequent labeling. The two-dimensional coordinates of the lane lines are obtained from the target point cloud image. This process simplifies lane line labeling to two-dimensional labeling, which is more efficient and less costly than directly labeling three-dimensional coordinates. Based on the first two-dimensional coordinates of the lane lines and the stitched point cloud, the first three-dimensional coordinates of the lane lines are determined. This means that the stitched point cloud automatically expands the already labeled two-dimensional coordinates into three-dimensional coordinates, significantly improving the efficiency of lane line labeling. In other words, the technical solution provided in this application essentially obtains the three-dimensional coordinates of the lane lines indirectly through their two-dimensional coordinates, thereby completing the three-dimensional labeling of the lane lines. Compared to directly labeling the three-dimensional coordinates of lane lines, the method in this application improves the efficiency and reduces the cost of lane line labeling.

[0026] When annotating lane lines in a target point cloud image, the main method used is to use preset annotation tools. This reduces the development cost of lane line annotation and improves the annotation speed by using a two-dimensional method.

[0027] Specifically, a method for determining the first three-dimensional coordinates of lane lines is proposed. When labeling lane lines in a target point cloud image, the pixel coordinates (i.e., the first pixel coordinates) of the lane lines are generally labeled. However, the first pixel coordinates are not coordinates on the physical imaging plane. Therefore, it is first necessary to convert the first pixel coordinates into coordinates on the physical imaging plane, i.e., the image coordinates of the lane lines in the target point cloud image. Further, based on this, and combined with the stitched point cloud, the first three-dimensional coordinates of the lane lines in the stitched point cloud coordinate system can be obtained. Through the above process, the three-dimensional lane line coordinates can be obtained from the two-dimensional coordinates of the lane lines, thus transforming the lane line labeling problem from two-dimensional labeling to three-dimensional labeling, completing the process of labeling the three-dimensional coordinates of the lane lines.

[0028] Specifically, this paper proposes a method to obtain the first three-dimensional coordinates of lane lines based on the image coordinates of lane lines in a target point cloud image and the stitched point cloud. Since the target point cloud image is a top view of the stitched point cloud, this application can determine the position of each point in the stitched point cloud coordinate system based on the image coordinates of each point of the lane line in the target point cloud image. Furthermore, based on the three-dimensional coordinates of each point in the stitched point cloud coordinate system, the first three-dimensional coordinates of the lane line in the stitched point cloud coordinate system are obtained. Through the above process, in the process of annotating lane lines, the three-dimensional coordinates of the lane lines can be obtained by combining the two-dimensional coordinates of the lane lines with the height of each point cloud, thus realizing the conversion from two-dimensional annotation to three-dimensional annotation, thereby reducing the difficulty of lane line annotation and making the lane line annotation process simpler and more efficient.

[0029] In addition to point cloud acquisition equipment, vehicles typically also include cameras. Therefore, in this application, the obtained first three-dimensional coordinates can also be projected onto the camera image plane.

[0030] The first three-dimensional coordinates are the three-dimensional coordinates in the stitched point cloud coordinate system. Since the stitched point cloud is composed of multiple different point clouds from a LiDAR scanner, it is not the original multi-frame point cloud. Therefore, after obtaining the three-dimensional coordinates in the stitched point cloud coordinate system, this application needs to restore the first three-dimensional coordinates one by one, that is, transform each three-dimensional coordinate in the stitched point cloud to the original point cloud coordinate system to obtain the original lane line's second three-dimensional coordinates. Further, based on the second three-dimensional coordinates, each frame is projected onto the camera image plane to obtain the second two-dimensional coordinates in the camera image plane.

[0031] If there are a large number of vehicle cameras, the above process can project a single point cloud image onto multiple camera image planes at once, thereby obtaining the two-dimensional coordinates of lane lines in multiple camera image planes simultaneously and rapidly improving the efficiency of lane line labeling.

[0032] Furthermore, a specific process for determining the second two-dimensional coordinates of the lane lines in the camera image plane is proposed. These second two-dimensional coordinates are the second pixel coordinates in the camera image plane. First, the second three-dimensional coordinates of the lane lines are converted into third three-dimensional coordinates in the camera coordinate system. This process transforms the lane line coordinates from the original point cloud coordinate system to the camera coordinate system, thus achieving projection of the lane lines onto the camera plane based on point cloud data. Furthermore, in camera imaging, the image captured by the camera is ultimately on the imaging plane. Therefore, it is necessary to further combine the coordinate transformation relationship between camera coordinates and image coordinates to obtain the second pixel coordinates of the lane lines in the camera image plane, thereby completing the projection of the lane lines onto the camera plane.

[0033] When there are a large number of vehicle cameras in the above projection process, based on a single frame of original point cloud, the intrinsic parameter matrices of each of the multiple cameras, and the transformation relationship between coordinates, the embodiments of this application can project onto the image planes of multiple cameras at once, thereby obtaining the two-dimensional coordinates of lane lines in the image planes of multiple cameras simultaneously, which greatly improves the efficiency of lane line labeling.

[0034] In the process of converting the stitched point cloud into a target point cloud image, a voxelization method is specifically used. First, the stitched point cloud is divided into at least one voxel. Then, the target point cloud image is obtained by projecting each voxel onto the plane containing the target point cloud image.

[0035] The stitched point cloud is obtained by stitching together multiple point clouds. In this application, based on the principle of point cloud stitching, the multiple point clouds are rotated and translated during the stitching process to unify them into the same coordinate system, thus obtaining the stitched point cloud, which provides a basis for subsequent lane line annotation based on the multi-frame point cloud.

[0036] Secondly, a device for marking lane lines is provided. The device includes: an acquisition module for acquiring a stitched point cloud on a road where a vehicle is traveling, the stitched point cloud being obtained by stitching together multiple frames of point clouds collected when the vehicle is traveling on the road; a spatial transformation module for performing a spatial transformation on the stitched point cloud to obtain a target point cloud image, the target point cloud image being a projection image of the stitched point cloud on the ground; and a first determination module for determining the first two-dimensional coordinates of the lane line in the target point cloud image; and determining the first three-dimensional coordinates of the lane line based on the first two-dimensional coordinates of the lane line and the stitched point cloud.

[0037] In conjunction with the second aspect, in some possible implementations, the first determining module is specifically used to: annotate the target point cloud image using a preset annotation tool to obtain the first two-dimensional coordinates of the lane line.

[0038] In combination with the second aspect and the above implementation methods, in some possible implementation methods, the first two-dimensional coordinates are the first pixel coordinates of the lane line in the target point cloud image, and the first determining module is further configured to: determine the image coordinates of the lane line in the target point cloud image according to the first coordinate transformation relationship and the first pixel coordinates of the lane line; and determine the first three-dimensional coordinates of the lane line in the stitched point cloud coordinate system according to the image coordinates of the lane line and the stitched point cloud.

[0039] In conjunction with the second aspect and the above implementation methods, in some possible implementation methods, the first determining module is further configured to: for any point on the lane line, determine the height of the point in the stitched point cloud based on the image coordinates of the point; and determine the first three-dimensional coordinates of the lane line based on the image coordinates of the point and the height of the point in the stitched point cloud.

[0040] In conjunction with the second aspect and the above implementation methods, in some possible implementation methods, after determining the first three-dimensional coordinates of the lane line in the stitched point cloud coordinate system based on the image coordinates and the stitched point cloud, the device further includes: a second determining module, used to determine the second three-dimensional coordinates of the lane line in the original point cloud coordinate system corresponding to the multi-frame point cloud based on the second coordinate transformation relationship and the first three-dimensional coordinates of the lane line; and to determine the second two-dimensional coordinates of the lane line in the camera image plane of the vehicle based on the third coordinate transformation relationship and the second three-dimensional coordinates of the lane line.

[0041] In combination with the second aspect and the above implementation, in some possible implementations, the second two-dimensional coordinate is the second pixel coordinate of the lane line in the camera image plane. The second determining module is specifically used to: determine the third three-dimensional coordinate of the lane line in the camera coordinate system according to the fourth coordinate transformation relationship and the second three-dimensional coordinate of the lane line; and determine the second pixel coordinate of the lane line according to the camera's intrinsic parameter matrix and the fifth coordinate transformation relationship.

[0042] In combination with the second aspect and the above implementation methods, in some possible implementation methods, the spatial transformation module is specifically used to: segment the stitched point cloud into at least one voxel; and project each voxel in the at least one voxel as a point to obtain the target point cloud image.

[0043] Combining the second aspect and the above implementation methods, in some possible implementation methods, the acquisition module is specifically used to: perform rotation and / or translation processing on the multi-frame point cloud based on the point cloud registration algorithm to obtain the stitched point cloud.

[0044] Thirdly, an electronic device is provided, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, causing the electronic device to perform the methods of the first aspect or any possible implementation thereof.

[0045] Fourthly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0046] Fifthly, a computer-readable storage medium is provided that stores computer program code, which, when executed on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of a scenario for marking lane lines provided in an embodiment of this application;

[0048] Figure 2 This is a schematic flowchart illustrating a method for marking lane lines provided in an embodiment of this application;

[0049] Figure 3 This is a scene diagram illustrating a pixel coordinate system and an image coordinate system provided in an embodiment of this application;

[0050] Figure 4 This is a scene diagram of four coordinate systems in a camera provided in an embodiment of this application;

[0051] Figure 5 This is a schematic diagram of the structure of a lane marking device provided in an embodiment of this application;

[0052] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0054] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0055] Figure 1 This is a schematic diagram of a scenario for marking lane lines provided in an embodiment of this application.

[0056] For example, such as Figure 1 As shown, in order to ensure the safety of vehicle 101 while it is traveling on road 102, it is necessary to determine the relative positional relationship between vehicle 101 and lane 103 so that vehicle 101 can travel within the safe area of ​​lane 103.

[0057] In one possible implementation, lane lines can be marked and identified in advance, so that when vehicle 101 is traveling on road 102, it can determine whether it has deviated from lane 103 based on the current lane 103 and its own position.

[0058] The labeling and identification of lane lines can specifically include: labeling the category attributes of lane lines, such as dashed lines, solid lines, double dashed lines, etc.; the color attributes of lane lines, such as white, yellow, green, blue, etc.; lane definition attributes, such as the left lane line of the main lane, the right lane line of the main lane, the left lane line of the adjacent lane, the right lane line of the adjacent lane, etc.; lane line occlusion attributes, such as no occlusion, slight occlusion, moderate occlusion, severe occlusion, and the coordinates of the lane lines, etc.

[0059] Lane marking is a crucial prerequisite for Vehicle 101 to achieve intelligent driving. After lane marking is completed, Vehicle 101 can perform the following functions:

[0060] Lane Departure Warning (LDW) is a feature used in autonomous driving scenarios. Specifically, when vehicle 101 deviates from lane 103, the LDW system can obtain the deviation information of vehicle 101 and remind the driver through sound, touch, and other means to avoid dangerous accidents after vehicle 101 crosses the line or lane.

[0061] Lane Keeping Assist (LKA) is a feature that, compared to Lane Keeping Ward (LDW), can actively control the steering wheel of vehicle 101 to correct the lateral position of vehicle 101 and bring vehicle 101 back into lane 103 when vehicle 101 deviates from lane 103.

[0062] Lane Centering Control (LCC) is a function that helps the driver control the steering wheel of vehicle 101 to center the vehicle 101 in the center of lane 103 and continuously control the vehicle 101 to drive in the center of lane 103, freeing the driver's hands.

[0063] Of course, in addition to the common lane line recognition applications mentioned above, lane lines can also be applied to a large number of autonomous driving scenarios, which will not be listed one by one in this application.

[0064] After introducing the lane marking and application scenarios, the following describes a method for marking lane lines provided by an embodiment of this application.

[0065] Figure 2 This is a schematic flowchart illustrating a method for marking lane lines provided in an embodiment of this application.

[0066] For example, such as Figure 2 As shown, the method 200 includes:

[0067] 201. Obtain the stitched point cloud on the road where the vehicle is traveling. The stitched point cloud is obtained by stitching together multiple frames of point cloud collected when the vehicle is traveling on the road.

[0068] It should be understood that, in this embodiment of the application, the vehicle can generally be equipped with a higher-precision point cloud acquisition device. The point cloud acquisition device can collect three-dimensional point cloud data on the road surface where the vehicle is traveling. Therefore, this embodiment of the application can utilize the three-dimensional point cloud data collected by the point cloud acquisition device to complete the lane marking process.

[0069] Optionally, the point cloud acquisition device includes any device capable of acquiring three-dimensional point cloud data, such as mechanical lidar, solid-state lidar, binocular camera, multi-view camera, contact scanner, structured light, triangulation, etc. This application embodiment does not limit the type of point cloud acquisition device.

[0070] Specifically, the stitched point cloud is obtained by stitching together multiple frames of point cloud (also known as multiple frames of original point cloud or original point cloud) acquired by a point cloud acquisition device. Therefore, the embodiments of this application first need to acquire multiple frames of original point cloud and then stitch them together to obtain the stitched point cloud.

[0071] In one possible implementation, the process of obtaining the stitched point cloud specifically includes:

[0072] Based on the point cloud registration algorithm, the poses of multiple frames of point clouds are rotated and / or translated to obtain a stitched point cloud.

[0073] It should be understood that, taking LiDAR as an example of a point cloud acquisition device, when scanning the environment, LiDAR often cannot measure the point cloud data of the environment in one coordinate system at the same time. This is because the size of the environment may exceed the measurement range of the LiDAR, and it is unlikely to scan the complete point cloud of an object from a single angle. Therefore, it is necessary to scan from multiple angles. After obtaining multiple frames of point cloud data from multiple angles, in order to obtain complete environmental point cloud data, it is necessary to rotate and translate the multiple frames of point cloud data to a unified coordinate system to obtain a stitched point cloud.

[0074] When acquiring multiple frames of point cloud data, the detection range of the point cloud acquisition device can generally be preset. Typically, the detection range of a point cloud acquisition device is approximately 200-300 meters. A single frame of the original point cloud usually contains a large number of isolated points.

[0075] For example, during the rotation and translation of multiple point clouds, this can be achieved by acquiring the pose of each point cloud in the multiple frames. The pose of the multiple point clouds can be obtained by locating the point cloud acquisition device using a positioning system in Simultaneous Localization and Mapping (SLAM) or Real-Time Kinematic (RTK, also known as carrier phase differential) technology. For instance, when the point cloud acquisition device acquires each frame of the point cloud, the positioning system can be used to locate the point cloud acquisition device, and the pose of the point cloud acquisition device can be used as the pose of each frame of the point cloud.

[0076] Optionally, the positioning system can be the Global Positioning System (GPS) or the Global Navigation Satellite System (GNSS), but this application does not limit this.

[0077] For example, consider RTK (Real-Time Kinematics). An RTK system includes a base station, a rover (mobile station, user station, or vehicle in this embodiment), and a data communication link (e.g., a mobile network). Both the base station and the rover are equipped with positioning systems. The base station knows its own three-dimensional coordinates, which are typically fixed. The GPS in the base station acquires a first carrier phase signal and transmits the first carrier phase signal and its own three-dimensional coordinates to the rover via the data communication link. The GPS in the point cloud acquisition device in the vehicle can also acquire a second carrier phase signal.

[0078] Furthermore, in this embodiment, the relative position (relative three-dimensional coordinates) between the point cloud acquisition device and the base station can be determined based on dynamic differential positioning. Then, the pose of each frame of point cloud can be determined using the relative three-dimensional coordinates and the three-dimensional coordinates of the base station.

[0079] It should be understood that the vehicle's position and angle constantly change during travel, and the point cloud acquisition device has a limited acquisition range. Therefore, point cloud acquisition devices often cannot perform a single measurement of the point cloud of the driving road in the same coordinate system. The coordinate systems of each frame of the point cloud in a multi-frame point cloud are different. In order to simultaneously annotate the lane lines in multiple frames of point clouds, the embodiments of this application need to first stitch the multiple frames of point clouds together to obtain a complete stitched point cloud (or global point cloud) within a preset range in the same coordinate system.

[0080] Point cloud stitching is the process of registering overlapping portions of point clouds at any location; it is also called point cloud registration. Depending on the registration method, point cloud registration can be divided into rigid body registration and non-rigid body registration. Rigid body registration involves only spatial rotation and translation transformations; non-rigid body registration involves scaling, deformation, and affine transformations. The point cloud registration process in this application primarily refers to rigid body registration.

[0081] For example, the rigid body registration process mainly involves determining the rotation matrix (R) and translation matrix (T) between adjacent point clouds based on the poses of multiple point clouds. Based on the registration between adjacent point clouds, a stitched point cloud can be obtained after the multi-frame point cloud stitching process.

[0082] Optionally, point cloud stitching algorithms include: Point Feature Histogram (PFH), Viewpoint Feature Histogram (VFH), Point Pair Feature (PPF), Normal Distributions Transform (NDT) algorithm, and Iterative Closest Point (ICP) algorithm. This application does not limit these algorithms.

[0083] In the above technical solution, the stitched point cloud is obtained by stitching together multiple frames of point clouds. Specifically, in the stitching process of this application, based on the principle of point cloud stitching, the multiple frames of point clouds are rotated and translated to unify them into the same coordinate system, thus obtaining the stitched point cloud, which provides a basis for subsequent lane line annotation based on the multiple frames of point clouds.

[0084] Furthermore, during the sampling process, the original point cloud may contain isolated points, noise, and other issues, and the large amount of point cloud data makes it difficult to process. Therefore, after obtaining the stitched point cloud, filtering and downsampling processing can be performed on the point cloud.

[0085] Alternatively, filtering and downsampling can be performed on each frame of the original point cloud after it has been obtained but before it has been stitched together. This application does not limit this approach.

[0086] Optionally, point cloud filtering algorithms include pass-through filters, voxel filters, statistical filters, conditional filters, radius filters, bilateral filters, Gaussian filters, uniform sampling filters, moving least squares filters, frequency filters, etc., and the embodiments of this application do not limit them.

[0087] Optionally, point cloud downsampling methods include voxel downsampling, uniform downsampling, curvature downsampling, surface uniform downsampling, Poisson disk sampling, etc., and this application embodiment does not limit these methods.

[0088] 202. Perform spatial transformation on the stitched point cloud to obtain the target point cloud image, which is the projection image of the stitched point cloud onto the ground.

[0089] The target point cloud image is a bird's-eye view (BEV) of the stitched point cloud.

[0090] It should be understood that, since BEV (Blocked Elevated Point Cloud) can provide a more comprehensive perception perspective, the embodiments of this application can project the stitched point cloud after obtaining a stitched point cloud from multiple frames to obtain its projected image on the ground plane, i.e., the target point cloud image, also known as the "stitched point cloud BEV image". Thus, through the above projection process, dimensionality reduction of the point cloud can be achieved, reducing the workload of subsequent annotation.

[0091] One possible implementation involves performing spatial transformations on the stitched point cloud to obtain the target point cloud image, specifically including:

[0092] Divide the stitched point cloud into at least one voxel;

[0093] Projecting each voxel in at least one voxel as a point yields the target point cloud image.

[0094] For example, in this embodiment, a region of interest can be first set on the acquired stitched point cloud, and then feature extraction can be performed based on the points. The point cloud can be divided into blocks using small cubes of the same size, each of which can be considered a voxel. Furthermore, voxel features of each voxel can be extracted using a model, such as the VoxelNet model. Finally, the voxel features are mapped to the corresponding positions in the target point cloud image to obtain the target point cloud image.

[0095] In the above technical solution, voxelization is specifically used in the process of converting the stitched point cloud into a target point cloud image. First, the stitched point cloud is divided into at least one voxel. Then, the target point cloud image is obtained by projecting each voxel onto the plane containing the target point cloud image.

[0096] 203. Determine the first two-dimensional coordinates of the lane lines in the target point cloud image.

[0097] In this step, the target point cloud image can be directly imported into the preset annotation tool using a preset standard tool, which will automatically annotate the first two-dimensional coordinates of the lane lines in the target point cloud image.

[0098] Optionally, the default standard tool is an image annotation tool, such as Labelme, Computer Vision Annotation Tool (CVAT), Visual Object Tagging Tool (VOTT), Labelme, Visual Geometry Group Image Annotator (VIA), Pixel Annotation Tool (PAT), etc., and this application embodiment does not limit it.

[0099] For example, using Labelme, the process of annotating lane lines can be divided into the following steps: 1. Open the Labelme software in the interface; 2. After entering the interface, select the file corresponding to the target point cloud image, and mark the lane lines using the "Create LineStrip" or "Create Polygons" option; 3. Batch convert the JavaScript object notation (JSON) files to obtain the target point cloud image of the annotated lane lines.

[0100] In the process of labeling lane lines, they can be labeled in the form of lines or polygons. The main labels are the category attributes, color attributes, lane definition attributes, and lane line occlusion attributes of the lane lines.

[0101] Optionally, the lane line category attribute can be no lane line, dashed line, solid line, double dashed line, double solid line, dashed and solid line, etc.; the lane line color attribute can be white, yellow, green, blue, red, etc.; the lane line definition attribute can be the lane line to the left of the main lane line, the lane line to the right of the main lane line, the lane line to the right of the adjacent lane, the grid line to the left, the grid line to the right, the intersection to the left, the intersection to the right, etc.

[0102] In the above technical solution, when annotating lane lines in the target point cloud image, the main method used is to use a preset annotation tool. On the one hand, this can reduce the development cost in the lane line annotation process, and on the other hand, annotating lane lines in a two-dimensional way can improve the annotation speed.

[0103] 204. Based on the first two-dimensional coordinates of the lane line and the spliced ​​point cloud, determine the first three-dimensional coordinates of the lane line.

[0104] It should be understood that after lane lines are labeled in a target point cloud image, the resulting data is typically the pixel coordinates of the lane lines. In other words, the first two-dimensional coordinates of the lane lines in this embodiment represent the first pixel coordinates of the lane lines in the target point cloud image. For clarity, this embodiment uses (u1, v1) to represent the first pixel coordinates.

[0105] Furthermore, since the target point cloud image is obtained by stitching together point cloud projections, the embodiments of this application can perform coordinate transformation based on the first two-dimensional coordinates of the lane lines to obtain the first three-dimensional coordinates of the lane lines in the stitched point cloud coordinate system.

[0106] In one possible implementation, obtaining the first three-dimensional coordinates of the lane line based on its first two-dimensional coordinates includes:

[0107] Based on the first coordinate transformation relationship and the first pixel coordinates of the lane line, determine the image coordinates of the lane line in the target point cloud image;

[0108] Based on the image coordinates of the lane lines and the stitched point cloud, determine the first three-dimensional coordinates of the lane lines in the stitched point cloud coordinate system.

[0109] It should be understood that a target point cloud image is an image composed of individual pixels. After annotation, the first pixel coordinates of the lane lines in the target point cloud image can be directly obtained. However, the first pixel coordinates are not coordinates on the physical imaging plane and cannot be directly used for the reconstruction of three-dimensional coordinates. Therefore, this embodiment first needs to convert the first two-dimensional coordinates of the lane lines into image coordinates of the lane lines in the target point cloud image, that is, coordinates on the physical imaging plane.

[0110] Figure 3This is a scene diagram of a pixel coordinate system and an image coordinate system provided in an embodiment of this application.

[0111] For example, such as Figure 3 As shown, O0 is the origin of the pixel coordinate system (also called the image pixel coordinate system). Generally, the origin of the pixel coordinate system is located in the upper left of the image plane. The two coordinate axes of the pixel coordinate system are represented by the u-axis and v-axis. The u-axis is parallel to the image plane and points horizontally to the right, while the v-axis is perpendicular to the u-axis and points downwards, as shown below. Figure 3 As shown. In the pixel coordinate system, each coordinate point can be represented by (u, v). Coordinate points in the pixel coordinate system have no physical units.

[0112] O1 is the origin of the image coordinate system. Generally, the origin of the image coordinate system (also called the physical image coordinate system) is located at the center of the pixel coordinate system. The two coordinate axes of the image coordinate system are represented by the x-axis and y-axis. The x-axis is parallel to the u-axis, and the y-axis is parallel to the v-axis. In the image coordinate system, each coordinate point can be represented as (x, y). The physical unit of coordinate points in the image coordinate system is millimeters (mm).

[0113] The transformation relationship between image coordinates and pixel coordinates can be expressed by the following formula (1):

[0114]

[0115] In formula (1):

[0116] u0: The x-coordinate of the origin O1 of the image coordinate system in the pixel coordinate system (i.e., the coordinate in the u-axis direction);

[0117] v0: The ordinate of the origin O1 of the image coordinate system in the pixel coordinate system (i.e., the coordinate in the v-axis direction).

[0118] d x : The physical size of each pixel on the x-axis of the image coordinate system in the u-axis direction, in mm / pixel;

[0119] d y : The physical size of each pixel on the y-axis of the image coordinate system in the v-axis direction, in mm / pixel.

[0120] Furthermore, the above transformation relationship can be expressed in matrix form to obtain the coordinate transformation relationship between pixel coordinates and image coordinates, which can be represented by the following formula (2) or formula (3).

[0121]

[0122]

[0123] Formula (3) is the inverse transformation matrix of Formula (2).

[0124] The coordinate transformation relationship described above is the first coordinate transformation relationship. Using this coordinate transformation relationship and the first pixel coordinates, the image coordinates of the lane lines in the target point cloud image are obtained.

[0125] It should be understood that since the image coordinates only contain the coordinate values ​​of the x-axis and y-axis, in order to determine the first three-dimensional coordinates of the lane line in the stitched point cloud coordinate system, it is also necessary to combine the value of the lane line in the height coordinate (i.e., the value of the z-axis).

[0126] The above technical solution proposes a method for determining the first three-dimensional coordinates of lane lines. When labeling lane lines in a target point cloud image, the pixel coordinates (i.e., the first pixel coordinates) of the lane lines are generally labeled. However, the first pixel coordinates are not coordinates on the physical imaging plane. Therefore, it is first necessary to convert the first pixel coordinates into coordinates on the physical imaging plane, i.e., the image coordinates of the lane lines in the target point cloud image. Further, based on this, and combined with the stitched point cloud, the first three-dimensional coordinates of the lane lines in the stitched point cloud coordinate system can be obtained. Through the above process, the three-dimensional lane line coordinates can be obtained from the two-dimensional coordinates of the lane lines, thus transforming the lane line labeling problem from two-dimensional labeling to three-dimensional labeling, completing the process of labeling the three-dimensional coordinates of the lane lines.

[0127] Specifically, based on the image coordinates of the lane lines and the stitched point cloud, the first three-dimensional coordinates of the lane lines in the stitched point cloud coordinate system are obtained, including:

[0128] For any point on the lane line, determine the height of that point in the stitched point cloud based on its image coordinates.

[0129] The first three-dimensional coordinates of the lane line are determined based on the image coordinates of any point and the height of any point in the stitched point cloud.

[0130] For example, since the target point cloud image is obtained by projecting a stitched point cloud, for each coordinate point in the image coordinate system, its position (coordinates) in the stitched point cloud can be determined according to its position in the image coordinate system. Thus, the first three-dimensional coordinates of each coordinate point in the stitched point cloud coordinate system are obtained, denoted by (x1, y1, z1).

[0131] The above technical solution proposes a method to obtain the first three-dimensional coordinates of lane lines based on the image coordinates of lane lines in a target point cloud image and the stitched point cloud. Since the target point cloud image is a top view of the stitched point cloud, this application can determine the position of each point in the stitched point cloud coordinate system based on the image coordinates of each point of the lane line in the target point cloud image. Furthermore, based on the three-dimensional coordinates of each point in the stitched point cloud coordinate system, the first three-dimensional coordinates of the lane line in the stitched point cloud coordinate system are obtained. Through this process, in the process of annotating lane lines, the three-dimensional coordinates of the lane lines can be obtained by combining the two-dimensional coordinates of the lane lines with the height of each point cloud, thus realizing the conversion from two-dimensional annotation to three-dimensional annotation, thereby reducing the difficulty of lane line annotation and making the lane line annotation process simpler and more efficient.

[0132] It should be understood that a stitched point cloud is obtained by stitching together multiple frames of original point clouds, and all point clouds in the stitched point cloud correspond to the same coordinate system. However, in the actual acquisition process, the coordinate system corresponding to each point cloud is different due to the different acquisition times. In this embodiment, in order to quickly perform lane line annotation, the coordinate systems of all point clouds are converted to the same coordinate system for annotation. After obtaining the first three-dimensional coordinates of the lane line in the stitched point cloud coordinate system, in order to obtain the coordinates of the lane line in the coordinate system corresponding to each frame of the original point cloud, it is also necessary to restore the first three-dimensional coordinates of the lane line frame by frame to each frame of the original point cloud to obtain the second three-dimensional coordinates of the lane line in the coordinate system of each original point cloud.

[0133] In one possible implementation, transforming the first three-dimensional coordinates of the lane lines from the stitched point cloud coordinate system to the original point cloud coordinate system specifically includes:

[0134] Based on the second coordinate transformation relationship and the first three-dimensional coordinates of the lane lines, the second three-dimensional coordinates of the lane lines in the original point cloud coordinate system corresponding to the multi-frame point cloud are determined.

[0135] For example, as mentioned above, the process of stitching together multiple point clouds to obtain a stitched point cloud is essentially a process of finding spatial transformations between different point clouds. These spatial transformations mainly refer to rotation and translation matrices. Conversely, after obtaining the first three-dimensional coordinates of each point cloud in the stitched point cloud coordinate system, the process of restoring the first three-dimensional coordinates of the lane lines in the stitched point cloud to each original point cloud frame by frame can be similar to the above process. That is, first determine the inverse of the rotation matrix and / or the inverse of the translation matrix corresponding to each point cloud during the stitching process, and then further filter out the point cloud corresponding to the lane line from all point clouds, as well as the inverse of the rotation matrix and / or the inverse of the translation matrix of the point cloud corresponding to the lane line. Finally, based on the first three-dimensional coordinates of the lane line and the corresponding inverse of the rotation matrix and / or the inverse of the translation matrix, the second three-dimensional coordinates of the lane line in each original point cloud frame are obtained in reverse.

[0136] Typically, a stitched point cloud contains tens to thousands of original point cloud frames. In this embodiment, the stitched point cloud can be restored to each original point cloud frame one by one based on the inverse of the rotation matrix and the inverse of the translation matrix between each original point cloud frame and the stitched point cloud. This yields the second three-dimensional coordinates of the lane lines in the coordinate system of each original point cloud frame, denoted as (x2, y2, z2). The inverse of the rotation matrix and the inverse of the translation matrix mentioned above constitute the second coordinate transformation relationship in this embodiment.

[0137] It should be understood that current vehicles, in addition to point cloud acquisition equipment, are usually equipped with cameras. In order to obtain the lane line coordinates in the form of an image on the camera plane, embodiments of this application can project the second three-dimensional coordinates of the lane lines in the form of point cloud onto the camera image plane to obtain the second two-dimensional coordinates of the lane lines in the camera image plane.

[0138] In one possible implementation, determining the second two-dimensional coordinates of the lane line in the camera image plane specifically includes:

[0139] Based on the third coordinate transformation relationship and the second three-dimensional coordinates of the lane lines, the second two-dimensional coordinates of the lane lines in the vehicle's camera image plane are determined.

[0140] The second two-dimensional coordinates of the lane line refer to the second pixel coordinates of the lane line in the camera image plane, denoted as (u2, v2). During the projection of these second three-dimensional coordinates onto the camera plane, based on the camera imaging principle, the lane line is reflected in the camera image through pixel coordinates. Therefore, when projecting the lane line from the original point cloud coordinate system onto the camera plane, the final result is the lane line coordinates in the pixel coordinate system.

[0141] In the above technical solutions, in addition to the point cloud acquisition equipment, the vehicle typically also includes a camera. Therefore, in this application, the obtained first three-dimensional coordinates can also be projected onto the camera image plane.

[0142] The first three-dimensional coordinates are the three-dimensional coordinates in the stitched point cloud coordinate system. Since the stitched point cloud is composed of multiple different point clouds from a LiDAR scanner, it is not the original multi-frame point cloud. Therefore, after obtaining the three-dimensional coordinates in the stitched point cloud coordinate system, this application needs to restore the first three-dimensional coordinates one by one, that is, transform each three-dimensional coordinate in the stitched point cloud to the original point cloud coordinate system to obtain the original lane line's second three-dimensional coordinates. Further, based on the second three-dimensional coordinates, each frame is projected onto the camera image plane to obtain the second two-dimensional coordinates in the camera image plane.

[0143] If there are a large number of vehicle cameras, the above process can project a single point cloud image onto multiple camera image planes at once, thereby obtaining the two-dimensional coordinates of lane lines in multiple camera image planes simultaneously and rapidly improving the efficiency of lane line labeling.

[0144] In one possible implementation, the second two-dimensional coordinates for determining the lane lines specifically include:

[0145] Based on the fourth coordinate transformation relationship and the second three-dimensional coordinates of the lane lines, determine the third three-dimensional coordinates of the lane lines in the camera coordinate system;

[0146] The second pixel coordinates of the lane line are determined based on the camera's intrinsic parameter matrix and the fifth coordinate transformation relationship.

[0147] Specifically, when transforming from the original point cloud coordinate system to the pixel coordinate system of the camera image plane, based on the principle of coordinate transformation, it is first necessary to transform the original point cloud coordinate system to the camera coordinate system, and then further transform it to the pixel coordinate system of the camera image plane based on the camera coordinate system to obtain the second pixel coordinates of the lane line. This completes the above-mentioned projection process from the original point cloud coordinate system to the camera image plane.

[0148] For example, let's take a LiDAR as the point cloud acquisition device. The original point cloud coordinate system mentioned above is the LiDAR coordinate system, which can also be defined as the world coordinate system. Furthermore, based on the transformation relationship between the world coordinate system and the camera coordinate system, we can obtain the third and third-dimensional coordinates of the lane line in the camera coordinate system, denoted as "(x3,y3,z3)".

[0149] Figure 4 This is a scene diagram of four coordinate systems in a camera provided in an embodiment of this application.

[0150] For example, such as Figure 4 As shown, O CLet x be the origin of the camera coordinate system. A point in the camera coordinate system can be denoted as (x...). c ,y c ,z c ), which is (x3, y3, z3) as mentioned above; O W Let x be the origin of the world coordinate system. A point in the world coordinate system can be denoted as (x...). w ,y w ,z w ), which is (x2, y2, z2) mentioned above.

[0151] The relationship between the four coordinate systems can be simply described as follows: the world coordinate system is transformed into the camera coordinate system through translation and rotation; the camera coordinate system is transformed into the image coordinate system through the triangle principle in the imaging model; and the image coordinate system is transformed into the pixel coordinate system through translation and scaling.

[0152] The coordinate transformation relationship between the world coordinate system and the camera coordinate system can be expressed by the following formula (4):

[0153]

[0154] In formula (4):

[0155] R: Represents the rotation matrix from the world coordinate system to the camera coordinate system;

[0156] T: represents the translation matrix from the world coordinate system to the camera coordinate system.

[0157] The rotation matrix and translation matrix are the extrinsic parameters of the camera, and can also be understood as the joint extrinsic parameters of the camera and the LiDAR.

[0158] Where: R = R1 * R2 * R3

[0159]

[0160] θ: The angle of rotation around the z-axis required when rotating from the world coordinate system to the camera coordinate system;

[0161] The angle of rotation required around the x-axis when rotating from the world coordinate system to the camera coordinate system;

[0162] ω: The angle of rotation around the y-axis required when rotating from the world coordinate system to the camera coordinate system.

[0163] The coordinate transformation relationship described above is the fourth coordinate transformation relationship in the embodiments of this application. Through the above coordinate transformation relationship, the third three-dimensional coordinates of the lane line in the camera coordinate system can be obtained.

[0164] Furthermore, the coordinate transformation relationship between the camera coordinate system and the pixel coordinate system (that is, the fifth coordinate transformation relationship) can be represented by the following formula (5). According to the following formula (5), the second pixel coordinate of the lane line in the camera image plane can be obtained.

[0165]

[0166] In formula (5):

[0167] f: The focal length of the camera;

[0168] z c :depth;

[0169] u0: The x-coordinate of the origin of the image coordinate system in the pixel coordinate system (i.e., the coordinate in the u-axis direction);

[0170] v0: The ordinate of the origin of the image coordinate system in the pixel coordinate system (i.e., the coordinate in the v-axis direction).

[0171] The intrinsic parameter matrix of the camera is denoted as "K".

[0172] By using the above method, each frame of the original point cloud can be projected onto the camera image plane to obtain the second pixel coordinates of the lane line in the camera image plane, denoted as "(u2,v2)".

[0173] In another scenario, embodiments of this application can also obtain the second pixel coordinates in the camera image plane in one step through the coordinate transformation relationship between world coordinates and pixel coordinates.

[0174] Specifically, the coordinate transformation relationship between world coordinates and pixel coordinates in the camera image plane can be expressed by the following formula (6):

[0175]

[0176] In another scenario, this embodiment can first obtain the image coordinates of the lane line in the camera image plane based on the coordinate transformation relationship between camera coordinates and image coordinates. Then, based on the coordinate transformation relationship between image coordinates and pixel coordinates, the second pixel coordinates of the lane line in the camera image plane can be obtained. In principle, all methods of converting from the world coordinate system to the pixel coordinate system can be applied to this embodiment.

[0177] It should be understood that there is usually more than one camera in a vehicle, which can perform different functions. By pre-calibrating the joint extrinsic parameters between the LiDAR and each camera in the vehicle, and then combining them with the intrinsic parameter matrix of each camera, it is possible to project the third and third-dimensional coordinates of the lane lines in a single frame of the original point cloud onto the image plane of all cameras at once.

[0178] The above technical solution specifically proposes a process for determining the second two-dimensional coordinates of the lane line in the camera image plane. Here, the second two-dimensional coordinates are the second pixel coordinates in the camera image plane. First, the second three-dimensional coordinates of the lane line are converted into the third three-dimensional coordinates of the lane line in the camera coordinate system. This process transforms the lane line coordinates from the original point cloud coordinate system to the camera coordinate system, thereby achieving projection of the lane line onto the camera plane based on point cloud data. Furthermore, in camera imaging, the image captured by the camera is ultimately on the imaging plane. Therefore, it is necessary to further combine the coordinate transformation relationship between camera coordinates and image coordinates to obtain the second pixel coordinates of the lane line in the camera image plane, thus completing the projection of the lane line onto the camera plane.

[0179] When there are a large number of vehicle cameras in the above projection process, based on a single frame of original point cloud, the intrinsic parameter matrices of each of the multiple cameras, and the transformation relationship between coordinates, the embodiments of this application can project onto the image planes of multiple cameras at once, thereby obtaining the two-dimensional coordinates of lane lines in the image planes of multiple cameras simultaneously, which greatly improves the efficiency of lane line labeling.

[0180] In summary, this application proposes a method for labeling lane lines. First, multiple frames of point clouds along the road are stitched together to obtain a stitched point cloud. This stitched point cloud is then projected to convert it into a target point cloud image, i.e., a top-down view of the stitched point cloud parallel to the ground, achieving dimensionality reduction and decreasing the workload for subsequent labeling. The two-dimensional coordinates of the lane lines are obtained from the target point cloud image. This process simplifies lane line labeling to two-dimensional labeling, which is more efficient and less costly than directly labeling three-dimensional coordinates. Based on the first two-dimensional coordinates of the lane lines and the stitched point cloud, the first three-dimensional coordinates of the lane lines are determined. This means that the stitched point cloud automatically expands the already labeled two-dimensional coordinates into three-dimensional coordinates, significantly improving the efficiency of lane line labeling. In other words, the technical solution provided in this application essentially obtains the three-dimensional coordinates of the lane lines indirectly through their two-dimensional coordinates, thereby completing the three-dimensional labeling of the lane lines. Compared to directly labeling the three-dimensional coordinates of lane lines, the method in this application improves the efficiency and reduces the cost of lane line labeling.

[0181] When annotating lane lines in a target point cloud image, the main method used is to use preset annotation tools. This reduces the development cost of lane line annotation and improves the annotation speed by using a two-dimensional method.

[0182] Specifically, a method for determining the first three-dimensional coordinates of lane lines is proposed. When labeling lane lines in a target point cloud image, the pixel coordinates (i.e., the first pixel coordinates) of the lane lines are generally labeled. However, the first pixel coordinates are not coordinates on the physical imaging plane. Therefore, it is first necessary to convert the first pixel coordinates into coordinates on the physical imaging plane, i.e., the image coordinates of the lane lines in the target point cloud image. Further, based on this, and combined with the stitched point cloud, the first three-dimensional coordinates of the lane lines in the stitched point cloud coordinate system can be obtained. Through the above process, the three-dimensional lane line coordinates can be obtained from the two-dimensional coordinates of the lane lines, thus transforming the lane line labeling problem from two-dimensional labeling to three-dimensional labeling, completing the process of labeling the three-dimensional coordinates of the lane lines.

[0183] Specifically, this paper proposes a method to obtain the first three-dimensional coordinates of lane lines based on the image coordinates of lane lines in a target point cloud image and the stitched point cloud. Since the target point cloud image is a top view of the stitched point cloud, this application can determine the position of each point in the stitched point cloud coordinate system based on the image coordinates of each point of the lane line in the target point cloud image. Furthermore, based on the three-dimensional coordinates of each point in the stitched point cloud coordinate system, the first three-dimensional coordinates of the lane line in the stitched point cloud coordinate system are obtained. Through the above process, in the process of annotating lane lines, the three-dimensional coordinates of the lane lines can be obtained by combining the two-dimensional coordinates of the lane lines with the height of each point cloud, thus realizing the conversion from two-dimensional annotation to three-dimensional annotation, thereby reducing the difficulty of lane line annotation and making the lane line annotation process simpler and more efficient.

[0184] In addition to point cloud acquisition equipment, vehicles typically also include cameras. Therefore, in this application, the obtained first three-dimensional coordinates can also be projected onto the camera image plane.

[0185] The first three-dimensional coordinates are the three-dimensional coordinates in the stitched point cloud coordinate system. Since the stitched point cloud is composed of multiple different point clouds from a LiDAR scanner, it is not the original multi-frame point cloud. Therefore, after obtaining the three-dimensional coordinates in the stitched point cloud coordinate system, this application needs to restore the first three-dimensional coordinates one by one, that is, transform each three-dimensional coordinate in the stitched point cloud to the original point cloud coordinate system to obtain the original lane line's second three-dimensional coordinates. Further, based on the second three-dimensional coordinates, each frame is projected onto the camera image plane to obtain the second two-dimensional coordinates in the camera image plane.

[0186] If there are a large number of vehicle cameras, the above process can project a single point cloud image onto multiple camera image planes at once, thereby obtaining the two-dimensional coordinates of lane lines in multiple camera image planes simultaneously and rapidly improving the efficiency of lane line labeling.

[0187] Furthermore, a specific process for determining the second two-dimensional coordinates of the lane lines in the camera image plane is proposed. These second two-dimensional coordinates are the second pixel coordinates in the camera image plane. First, the second three-dimensional coordinates of the lane lines are converted into third three-dimensional coordinates in the camera coordinate system. This process transforms the lane line coordinates from the original point cloud coordinate system to the camera coordinate system, thus achieving projection of the lane lines onto the camera plane based on point cloud data. Furthermore, in camera imaging, the image captured by the camera is ultimately on the imaging plane. Therefore, it is necessary to further combine the coordinate transformation relationship between camera coordinates and image coordinates to obtain the second pixel coordinates of the lane lines in the camera image plane, thereby completing the projection of the lane lines onto the camera plane.

[0188] When there are a large number of vehicle cameras in the above projection process, based on a single frame of original point cloud, the intrinsic parameter matrices of each of the multiple cameras, and the transformation relationship between coordinates, the embodiments of this application can project onto the image planes of multiple cameras at once, thereby obtaining the two-dimensional coordinates of lane lines in the image planes of multiple cameras simultaneously, which greatly improves the efficiency of lane line labeling.

[0189] In the process of converting the stitched point cloud into a target point cloud image, a voxelization method is specifically used. First, the stitched point cloud is divided into at least one voxel. Then, the target point cloud image is obtained by projecting each voxel onto the plane containing the target point cloud image.

[0190] The stitched point cloud is obtained by stitching together multiple point clouds. In this application, based on the principle of point cloud stitching, the multiple point clouds are rotated and translated during the stitching process to unify them into the same coordinate system, thus obtaining the stitched point cloud, which provides a basis for subsequent lane line annotation based on the multi-frame point cloud.

[0191] Figure 5 This is a schematic diagram of a device for marking lane lines provided in an embodiment of this application.

[0192] For example, such as Figure 5 As shown, the device 500 includes:

[0193] The acquisition module 501 is used to acquire the stitched point cloud on the road on which the vehicle is traveling. The stitched point cloud is obtained by stitching together multiple frames of point cloud collected when the vehicle is traveling on the road.

[0194] The spatial transformation module 502 is used to perform spatial transformation on the stitched point cloud to obtain a target point cloud image, which is a projection image of the stitched point cloud on the ground.

[0195] The first determining module 503 is used to determine the first two-dimensional coordinates of the lane line in the target point cloud image; and to determine the first three-dimensional coordinates of the lane line based on the first two-dimensional coordinates of the lane line and the stitched point cloud.

[0196] In one possible implementation, the first determining module 503 is specifically used to: annotate the target point cloud image using a preset annotation tool to obtain the first two-dimensional coordinates of the lane line.

[0197] In one possible implementation, the first two-dimensional coordinate is the first pixel coordinate of the lane line in the target point cloud image. The first determining module 503 is further configured to: determine the image coordinate of the lane line in the target point cloud image based on the first coordinate transformation relationship and the first pixel coordinate of the lane line; and determine the first three-dimensional coordinate of the lane line in the stitched point cloud coordinate system based on the image coordinate of the lane line and the stitched point cloud.

[0198] In one possible implementation, the first determining module 503 is further configured to: for any point on the lane line, determine the height of the point in the stitched point cloud based on the image coordinates of the point; and determine the first three-dimensional coordinates of the lane line based on the image coordinates of the point and the height of the point in the stitched point cloud.

[0199] Optionally, after determining the first three-dimensional coordinates of the lane line in the stitched point cloud coordinate system based on the image coordinates and the stitched point cloud, the device further includes: a second determining module, used to determine the second three-dimensional coordinates of the lane line in the original point cloud coordinate system corresponding to the multi-frame point cloud based on the second coordinate transformation relationship and the first three-dimensional coordinates of the lane line; and to determine the second two-dimensional coordinates of the lane line in the camera image plane of the vehicle based on the third coordinate transformation relationship and the second three-dimensional coordinates of the lane line.

[0200] In one possible implementation, the second two-dimensional coordinate is the second pixel coordinate of the lane line in the camera image plane. The second determining module is specifically used to: determine the third three-dimensional coordinate of the lane line in the camera coordinate system according to the fourth coordinate transformation relationship and the second three-dimensional coordinate of the lane line; and determine the second pixel coordinate of the lane line according to the camera's intrinsic parameter matrix and the fifth coordinate transformation relationship.

[0201] In one possible implementation, the spatial transformation module 502 is specifically used to: segment the stitched point cloud into at least one voxel; and project each voxel in the at least one voxel as a point to obtain the target point cloud image.

[0202] In one possible implementation, the acquisition module 501 is specifically used to: perform rotation and / or translation processing on the multi-frame point cloud based on the point cloud registration algorithm to obtain the stitched point cloud.

[0203] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0204] For example, such as Figure 6 As shown, the electronic device 600 includes a memory 601 and a processor 602. The memory 601 stores executable program code 6011, and the processor 602 is used to call and execute the executable program code 6011 to perform a method for marking lane lines.

[0205] This embodiment can divide the electronic device into functional modules according to the above method example. For example, each module can correspond to a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0206] When each functional module is divided according to its corresponding function, the electronic device may include: a unique acquisition module, a spatial transformation module, a first determination module, etc. It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0207] The electronic device provided in this embodiment is used to execute the above-described method for marking lane lines, and thus can achieve the same effect as the above-described implementation method.

[0208] When using integrated units, the electronic device may include a processing module and a storage module. The processing module is used to control and manage the operation of the electronic device. The storage module is used to support the execution of relevant program code and data by the electronic device.

[0209] The processing module may be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits shown in conjunction with the disclosure of this application. The processor may also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module may be a memory.

[0210] This embodiment also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the above-described method steps to implement a method for marking lane lines in the above embodiment.

[0211] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement a method for marking lane lines as described in the above embodiment.

[0212] In addition, the electronic device provided in the embodiments of this application may specifically be a chip, component or module. The electronic device may include a connected processor and a memory. The memory is used to store instructions. When the electronic device is running, the processor may call and execute the instructions to make the chip execute a method for marking lane lines in the above embodiments.

[0213] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0214] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0215] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0216] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for marking lane lines, characterized in that, The method includes: A stitched point cloud is obtained on the road on which the vehicle travels, wherein the stitched point cloud is obtained by stitching together multiple frames of point cloud collected when the vehicle travels on the road. A spatial transformation is performed on the stitched point cloud to obtain a target point cloud image, which is a projection image of the stitched point cloud on the ground. Determine the first two-dimensional coordinates of the lane lines in the target point cloud image; The first three-dimensional coordinates of the lane line are determined based on the first two-dimensional coordinates of the lane line and the spliced ​​point cloud.

2. The method according to claim 1, characterized in that, Determining the first two-dimensional coordinates of the lane line in the target point cloud image includes: The first two-dimensional coordinates of the lane line are obtained by annotating the target point cloud image using a preset annotation tool.

3. The method according to claim 1, characterized in that, The first two-dimensional coordinates are the first pixel coordinates of the lane line in the target point cloud image. Determining the first three-dimensional coordinates of the lane line based on the first two-dimensional coordinates of the lane line and the stitched point cloud includes: Based on the first coordinate transformation relationship and the first pixel coordinates of the lane line, the image coordinates of the lane line in the target point cloud image are determined; Based on the image coordinates of the lane line and the stitched point cloud, determine the first three-dimensional coordinates of the lane line in the stitched point cloud coordinate system.

4. The method according to claim 3, characterized in that, Determining the first three-dimensional coordinates of the lane line in the stitched point cloud coordinate system based on the image coordinates of the lane line and the stitched point cloud includes: For any point on the lane line, determine the height of the point in the stitching point cloud based on the image coordinates of the point. The first three-dimensional coordinates of the lane line are determined based on the image coordinates of any point and the height of any point in the stitched point cloud.

5. The method according to claim 3 or 4, characterized in that, After determining the first three-dimensional coordinates of the lane line in the stitched point cloud coordinate system based on the image coordinates and the stitched point cloud, the method further includes: Based on the second coordinate transformation relationship and the first three-dimensional coordinates of the lane line, determine the second three-dimensional coordinates of the lane line in the original point cloud coordinate system corresponding to the multi-frame point cloud; Based on the third coordinate transformation relationship and the second three-dimensional coordinates of the lane line, the second two-dimensional coordinates of the lane line in the camera image plane of the vehicle are determined.

6. The method according to claim 5, characterized in that, The second two-dimensional coordinates are the second pixel coordinates of the lane line in the camera image plane. Determining the second two-dimensional coordinates of the lane line in the camera image plane of the vehicle based on the third coordinate transformation relationship and the second three-dimensional coordinates of the lane line includes: Based on the fourth coordinate transformation relationship and the second three-dimensional coordinates of the lane line, determine the third three-dimensional coordinates of the lane line in the camera coordinate system; The second pixel coordinates of the lane line are determined based on the camera's intrinsic parameter matrix and the fifth coordinate transformation relationship.

7. The method according to claim 1, characterized in that, The step of performing spatial transformation on the stitched point cloud to obtain the target point cloud image includes: The stitched point cloud is segmented into at least one voxel; Projecting each voxel in the at least one voxel as a point yields the target point cloud image.

8. The method according to claim 1, characterized in that, The acquisition of the stitched point cloud along the vehicle's driving path includes: Based on the point cloud registration algorithm, the multi-frame point clouds are rotated and / or translated to obtain the stitched point cloud.

9. A device for marking lane lines, characterized in that, The device includes: The acquisition module is used to acquire the stitched point cloud on the road on which the vehicle is traveling. The stitched point cloud is obtained by stitching together multiple frames of point cloud collected when the vehicle is traveling on the road. A spatial transformation module is used to perform spatial transformation on the stitched point cloud to obtain a target point cloud image, wherein the target point cloud image is a projection image of the stitched point cloud on the ground. The first determining module is used to determine the first two-dimensional coordinates of the lane line in the target point cloud image; and to determine the first three-dimensional coordinates of the lane line based on the first two-dimensional coordinates of the lane line and the stitched point cloud.

10. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable program code; A processor for calling and running the executable program code from the memory, causing the electronic device to perform the method as described in any one of claims 1 to 8.

Images