Point cloud map establishment method, lane marking data acquisition method, device and medium

Abstract

The present invention relates to the field of autonomous driving technology, and specifically provides a point cloud map establishment method, a lane marking data acquisition method, a device, and a medium, aiming to solve the problem of improving the accuracy of lane marking data. To this end, the point cloud map establishment method provided by the present invention uses a global satellite positioning device to establish a three-dimensional point cloud map when the positioning signal of the global satellite positioning device is normal, so as to improve the accuracy of the three-dimensional point cloud map. When the positioning signal is abnormal, the relative position of the three-dimensional laser radar point cloud and the previous map is updated to improve the accuracy of the three-dimensional point cloud map; the lane marking data acquisition method provided by the present invention can use the above-mentioned point cloud map establishment method to establish a three-dimensional point cloud map, thereby improving the accuracy of the lane marking data.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for establishing a point cloud map, a method for acquiring lane marking data, a device, and a medium. Background Art

[0002] Autonomous driving technology mainly includes three key technologies: perception, planning, and control. Among them, perception technology is mainly used to determine the real-time local map and obstacle information of the driving environment in which the driving device (such as an autonomous driving vehicle) is located. Planning technology is mainly used to plan the driving trajectory of the driving device based on the above real-time local map and obstacle information. Control technology is mainly used to control the driving device to drive according to the planned driving trajectory.

[0003] To improve the accuracy of real-time local maps and obstacle information to ensure safe driving of the driving device, the current method is to first use lane annotation data of the lane scene to train a multi-sensor perception model based on the BEV (Bird Eye View) perspective, and then use the trained perception model to determine the real-time local map and obstacle information of the driving device's driving environment. The accuracy of the lane annotation data will greatly affect the accuracy of the perception model, and thus affect the accuracy of the real-time local map and obstacle information. Therefore, in order to improve the accuracy of the lane annotation data, improve the accuracy of the real-time local map and obstacle information, and thus ensure safe driving of the driving device, it is necessary to accurately obtain the lane annotation data of the lane scene.

[0004] Accordingly, this field requires a new technical solution to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned defects, the present invention is proposed to provide a point cloud map establishment method, lane marking data acquisition method, equipment and medium that solve or at least partially solve the technical problem of how to improve the accuracy of lane marking data, so as to improve the accuracy of real-time local maps and obstacle information, and thereby ensure that the driving device can drive safely.

[0006] In a first aspect, a method for establishing a three-dimensional point cloud map is provided, the method comprising:

[0007] Acquire the positioning signal of the global satellite positioning device on the vehicle during the vehicle's driving process and determine whether the positioning signal is normal; if normal, determine the radar positioning pose corresponding to each frame of the three-dimensional laser radar point cloud of the driving environment collected by the laser radar based on the posture parameters of the coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle and according to the global positioning pose obtained by the global satellite positioning device, splice each frame of the three-dimensional laser radar point cloud according to the radar positioning pose to establish a three-dimensional point cloud map of the driving environment; if abnormal, determine the relative pose of the current frame of the three-dimensional laser radar point cloud relative to the three-dimensional point cloud map established based on the previous multiple frames of three-dimensional laser radar point clouds for each frame of the three-dimensional laser radar point cloud, and update the three-dimensional point cloud map according to the relative pose and the current frame of the three-dimensional laser radar point cloud to establish a three-dimensional point cloud map of the driving environment.

[0008] In one technical solution of the above-mentioned method for establishing a three-dimensional point cloud map, the step of "determining the radar positioning pose corresponding to each frame of the three-dimensional lidar point cloud of the driving environment collected by the lidar based on the posture parameters of the coordinate system conversion between the global satellite positioning device and the lidar on the vehicle and according to the global positioning pose obtained by the global satellite positioning device" specifically includes: performing interpolation calculation on the global positioning pose obtained by the global satellite positioning device according to the timestamp corresponding to each frame of the three-dimensional lidar point cloud to obtain the global positioning pose at each said timestamp; determining the radar positioning pose corresponding to each frame of the three-dimensional lidar point cloud based on the posture parameters of the coordinate system conversion between the global satellite positioning device and the lidar and according to the global positioning pose at each said timestamp.

[0009] In one technical solution of the above-mentioned method for establishing a three-dimensional point cloud map, before the step of "determining the radar positioning pose corresponding to each frame of three-dimensional lidar point cloud of the driving environment collected by the lidar based on the pose parameters of the coordinate system conversion between the global satellite positioning device and the lidar on the vehicle and according to the global positioning pose obtained by the global satellite positioning device", the method also includes optimizing the pose parameters in the following manner: determining the radar positioning pose corresponding to each frame of three-dimensional lidar point cloud collected by the lidar when the vehicle turns based on the pose parameters and the global positioning pose obtained by the global satellite positioning device when the vehicle turns; and optimizing the pose parameters based on the radar positioning pose corresponding to each frame of three-dimensional lidar point cloud collected by the lidar when the vehicle turns using a point cloud registration algorithm based on the ICP (Iterative Closest Point) algorithm.

[0010] In one technical solution of the above-mentioned method for establishing a three-dimensional point cloud map, the step of "determining the relative pose of the current frame three-dimensional lidar point cloud relative to the three-dimensional point cloud map established based on the previous multiple frames of three-dimensional lidar point clouds" specifically includes: predicting the relative pose of the current frame three-dimensional lidar point cloud relative to the three-dimensional point cloud map established based on the previous multiple frames of three-dimensional lidar point clouds based on the angular velocity and linear velocity of the vehicle, and using the predicted relative pose as the prior relative pose; based on the prior relative pose, determining the relative pose of the current frame three-dimensional lidar point cloud relative to the three-dimensional point cloud map established based on the previous multiple frames of three-dimensional lidar point clouds.

[0011] In one technical solution of the above-mentioned method for establishing a three-dimensional point cloud map, after the step of “updating the three-dimensional point cloud map according to the relative pose and the current frame three-dimensional lidar point cloud to establish a three-dimensional point cloud map of the driving environment”, the method further includes: optimizing the radar positioning pose corresponding to the current frame three-dimensional lidar point cloud using a pose optimization method based on a factor graph (Pose Graph) according to the constraints of a preset factor graph (Pose Graph); wherein the constraints of the preset factor graph (Pose Graph) include radar positioning pose constraints and global positioning pose constraints;

[0012] The radar positioning posture constraint condition is as follows:

[0013]

[0014] Among them, r odom represents the constraint error of the radar positioning pose constraint condition, represents the absolute pose of the i-1th keyframe in the factor graph (PoseGraph), represents the absolute pose of the i-th key frame in the Pose Graph, Express Perform the inverse operation, represents the relative pose of the i-th keyframe relative to the i-1-th keyframe, w represents the world coordinate system, b represents the vehicle coordinate system, and the keyframe is a frame of 3D LiDAR point cloud whose radar positioning pose changes by more than a preset threshold compared to the previous frame of 3D LiDAR point cloud;

[0015] The global positioning pose constraint condition is as follows:

[0016]

[0017] Among them, r rtk represents the constraint error of the global positioning pose constraint, represents the global positioning pose at the timestamp corresponding to the j-th key frame in the factor graph (PoseGraph), T rb Indicates the pose parameters for the coordinate system conversion between the vehicle coordinate system and the global satellite positioning device. represents the absolute pose of the jth keyframe in the Pose Graph, Express An inverse operation is performed, where r represents the coordinate system of the global satellite positioning device, j∈Ω, and Ω represents the key frame in the three-dimensional lidar point cloud collected by the lidar when the positioning signal of the global satellite positioning device is normal.

[0018] In one technical solution of the above-mentioned method for establishing a three-dimensional point cloud map, before the step of "stitching each frame of three-dimensional lidar point cloud according to the radar positioning posture to establish a three-dimensional point cloud map of the driving environment", the method also includes performing point cloud dedistortion processing on each frame of three-dimensional lidar point cloud separately.

[0019] In one technical solution of the above-mentioned method for establishing a three-dimensional point cloud map, the step of "determining the relative pose of the current frame three-dimensional lidar point cloud relative to the three-dimensional point cloud map established based on the previous multiple frames of three-dimensional lidar point clouds" specifically includes: performing dynamic object detection on the current frame three-dimensional lidar point cloud, and removing the three-dimensional lidar point clouds belonging to dynamic objects in the current frame three-dimensional lidar point cloud according to the detection results; and determining the relative pose based on the current frame three-dimensional lidar point cloud after removing the three-dimensional lidar point clouds belonging to dynamic objects.

[0020] In a second aspect, a method for acquiring lane marking data is provided, the method comprising:

[0021] Acquire a three-dimensional lidar point cloud of the driving environment collected by a lidar on the vehicle during the vehicle's driving process; adopt the three-dimensional point cloud map establishment method described in any of the above technical solutions and establish a three-dimensional point cloud map of the driving environment based on the three-dimensional lidar point cloud; generate a bird's eye view (Bird Eye View) and a height map (Height Map) of the driving environment based on the three-dimensional point cloud map of the driving environment; and annotate lane elements of the driving environment during the vehicle's driving process based on the bird's eye view (Bird Eye View) and the height map (Height Map) to form four-dimensional lane element annotation data including one-dimensional time information and three-dimensional spatial position information.

[0022] In a third aspect, a computer device is provided, which includes a processor and a storage device, wherein the storage device is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the method for establishing a three-dimensional point cloud map or the method for obtaining lane marking data described in any one of the technical solutions of the above method.

[0023] In a fourth aspect, a computer-readable storage medium is provided, which stores a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the method for establishing a three-dimensional point cloud map or the method for obtaining lane marking data described in any one of the technical solutions of the above method.

[0024] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects:

[0025] In one technical solution implementing the present invention, a positioning signal from a global satellite positioning device on a vehicle can be obtained while the vehicle is in motion and its normality determined. When the positioning signal from the global satellite positioning device is normal, a three-dimensional point cloud map is created using the global satellite positioning device to improve the accuracy of the three-dimensional point cloud map. When the positioning signal from the global satellite positioning device is abnormal, the map is updated based on the relative position of the current frame's three-dimensional lidar point cloud and the previous map to improve the accuracy of the three-dimensional point cloud map. Furthermore, when the three-dimensional point cloud map is used for lane element annotation, the accuracy of lane element annotation can be significantly improved.

[0026] Specifically, if the positioning signal is normal, the radar positioning pose corresponding to each frame of the three-dimensional lidar point cloud of the driving environment collected by the lidar is determined based on the posture parameters of the coordinate system conversion between the global satellite positioning device and the lidar on the vehicle and the global positioning pose obtained by the global satellite positioning device, and each frame of the three-dimensional lidar point cloud is spliced according to the radar positioning pose to establish a three-dimensional point cloud map of the driving environment; if the positioning signal is abnormal, the relative posture of the current frame of the three-dimensional lidar point cloud relative to the three-dimensional point cloud map established based on the previous multiple frames of three-dimensional lidar point clouds is determined, and the three-dimensional point cloud map is updated according to the relative posture and the current frame of the three-dimensional lidar point cloud to establish a three-dimensional point cloud map of the driving environment.

[0027] In another technical solution for implementing the present invention, a three-dimensional lidar point cloud of the driving environment collected by a lidar on a vehicle during driving can be obtained, and then the aforementioned three-dimensional point cloud map establishment method is adopted and a three-dimensional point cloud map of the driving environment is established based on the three-dimensional lidar point cloud; a bird's-eye view (Bird Eye View) and a height map (Height Map) of the driving environment are generated respectively according to the three-dimensional point cloud map of the driving environment, and finally, the lane elements of the driving environment during vehicle driving are annotated according to the bird's-eye view and the height map to form four-dimensional lane element annotation data containing one-dimensional time information and three-dimensional spatial position information.

[0028] Through the above-mentioned implementation method, the annotation data of lane elements in four-dimensional space (a four-dimensional space formed by time and three-dimensional space) can be accurately obtained. After using these annotation data to train the vehicle's perception model, the perception ability of the perception model can be greatly improved, so that the real-time local map and obstacle information of the driving environment can be accurately determined during the vehicle's driving process, ensuring that the vehicle can drive safely. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. Those skilled in the art will readily appreciate that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Among them:

[0030] Figure 1 1 is a flow chart showing the main steps of a method for establishing a three-dimensional point cloud map according to an embodiment of the present invention;

[0031] Figure 2 This is a flow chart showing the main steps of a method for acquiring lane marking data according to one embodiment of the present invention;

[0032] Figure 3 1 is a flow chart showing the main steps of a method for removing a 3D lidar point cloud belonging to a dynamic object according to an embodiment of the present invention;

[0033] Figure 4 1 is a flow chart showing the main steps of a method for determining a ground plane of a three-dimensional point cloud map according to an embodiment of the present invention;

[0034] Figure 5 is a schematic diagram of a polar coordinate grid diagram of a three-dimensional point cloud map according to one embodiment of the present invention;

[0035] Figure 6 1 is a flow chart of the main steps of a method for calibrating pose parameters for coordinate system conversion between a laser radar and an image acquisition device according to an embodiment of the present invention;

[0036] Figure 7 It is a flowchart of the main steps of a method for calibrating posture parameters for coordinate system conversion between a laser radar and an image acquisition device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0037] Some embodiments of the present invention are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0038] In the description of the present invention, "processor" may include hardware, software, or a combination of the two. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware, or a combination of the two. Computer-readable storage media include any suitable medium that can store program code, such as a magnetic disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, etc. The term "A and / or B" represents all possible combinations of A and B, such as only A, only B, or A and B.

[0039] The following first describes an embodiment of a method for establishing a three-dimensional point cloud map.

[0040] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart showing the main steps of a method for establishing a three-dimensional point cloud map according to an embodiment of the present invention. Figure 1 As shown, the method for establishing a three-dimensional point cloud map in the embodiment of the present invention mainly includes the following steps S101 to S107.

[0041] Step S101: Acquire the positioning signal of the global satellite positioning device on the vehicle during the driving process of the vehicle. The global satellite positioning device refers to a device that uses satellite navigation positioning technology for positioning. In this embodiment, the global satellite positioning device can be a device that performs positioning based on the Global Navigation Satellite System (GNSS), a device that performs positioning based on the Global Positioning System (GPS), a device that performs positioning based on the BeiDou Navigation Satellite System (BDS), or a device based on RTK (Real Time Kinematic) positioning technology, etc. Those skilled in the art can flexibly select different types of global satellite positioning devices according to actual needs, and this embodiment does not specifically limit this.

[0042] Step S102: Determine whether the positioning signal is normal; if normal, execute steps S103 to S105; if abnormal, execute steps S106 to S107.

[0043] Step S103: Determine the pose parameters T for coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle go .

[0044] In practical applications, after installing a global satellite positioning device and a laser radar on a vehicle, it is necessary to convert the pose parameters T between the global satellite positioning device and the laser radar into coordinate systems. go In this embodiment, the position parameters T of the global satellite positioning device can be directly obtained. go The pose parameters T for coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle are obtained from the calibration results go .

[0045] Step S104: Based on the posture parameter T go According to the global positioning posture obtained by the global satellite positioning device, the radar positioning posture corresponding to each frame of the three-dimensional lidar point cloud of the driving environment collected by the lidar is determined respectively.

[0046] A 3D LiDAR point cloud is three-dimensional data determined by the echo signal reflected back to the LiDAR after receiving the electromagnetic waves sent by the vehicle's LiDAR. This 3D data contains the 3D coordinates of the environmental point in the point cloud coordinate system.

[0047] During vehicle driving, the vibration of the vehicle may cause a large error in the radar positioning posture actually output by the lidar. If the radar positioning posture actually output by the lidar is used to build a three-dimensional point cloud map of the driving environment, it will affect the accuracy of the three-dimensional point cloud map.

[0048] Based on the above posture parameter T go The determined radar positioning pose is not the actual radar positioning pose output by the lidar, but rather a radar positioning pose that matches the global positioning pose obtained through coordinate system conversion. Because global satellite positioning devices have relatively high positioning accuracy, the radar positioning pose matched using these pose parameters is also highly accurate. Therefore, using these pose parameters to create a 3D point cloud map of the driving environment greatly improves the accuracy of the 3D point cloud map.

[0049] Step S105: stitching each frame of the three-dimensional lidar point cloud according to the radar positioning posture to create a three-dimensional point cloud map of the driving environment.

[0050] Specifically, the relative poses of two adjacent frames of 3D lidar point clouds can be determined based on the radar positioning pose corresponding to each frame of 3D lidar point cloud, and then the two frames of 3D lidar point clouds can be matched based on the relative poses. The two frames of 3D lidar point clouds can be spliced together based on the results of the point cloud matching.

[0051] In some embodiments, in order to improve the accuracy of the three-dimensional point cloud map, each frame of the three-dimensional lidar point cloud can be first subjected to point cloud dedistortion processing, and then the three-dimensional lidar point cloud after point cloud dedistortion processing can be spliced to establish a three-dimensional point cloud map of the driving environment.

[0052] It should be noted that in this embodiment, the conventional point cloud dedistortion method in the field of three-dimensional laser radar point cloud technology can be used to perform point cloud dedistortion processing on each frame of three-dimensional laser radar point cloud respectively, which is not described in detail in this embodiment.

[0053] Step S106: for each frame of the 3D LiDAR point cloud, determine the relative pose (ego-motion) of the current frame of the 3D LiDAR point cloud relative to the 3D point cloud map established based on the previous multiple frames of the 3D LiDAR point cloud.

[0054] For the sake of simplicity, the "three-dimensional point cloud map established based on the previous multi-frame three-dimensional lidar point cloud" is referred to as the "prior map". In an embodiment of the present invention, the relative position of the current frame three-dimensional lidar point cloud and the prior map can be determined through the following steps 11 to 16.

[0055] Step 11: Filter out the first edge feature point and the first plane feature point from all three-dimensional lidar point clouds contained in the current frame three-dimensional lidar point cloud.

[0056] Specifically, the roughness of each three-dimensional lidar point cloud can be calculated, and the three-dimensional lidar point cloud with a roughness less than a preset roughness threshold can be used as the first plane feature point, and the three-dimensional lidar point cloud with a roughness greater than or equal to the preset roughness threshold can be used as the first edge feature point.

[0057] It should be noted that in this embodiment, the roughness of each 3D lidar point cloud can be calculated using conventional point cloud roughness calculation methods in the field of 3D point cloud processing technology, and this embodiment does not specifically limit this calculation method. In addition, those skilled in the art can flexibly set the specific value of the preset roughness threshold according to actual needs, and this embodiment also does not specifically limit this.

[0058] Step 12: Filter out the second edge feature points and the second plane feature points from the three-dimensional lidar point cloud contained in the previous map, and establish a KD-Tree (K-dimensional search tree) of the three-dimensional lidar point cloud based on the filtered second edge feature points and the second plane feature points.

[0059] The method for selecting the second edge feature points and the second plane feature points from the 3D LiDAR point cloud contained in the prior map is the same as the method for selecting the first edge feature points and the first plane feature points in step 11 above, and will not be repeated here. In addition, in this embodiment, a conventional KD-Tree establishment method can be used to establish a KD-Tree for the 3D LiDAR point cloud based on the selected second edge feature points and the second plane feature points.

[0060] Step 13: Based on the KD-Tree of the 3D lidar point cloud, search for the second edge feature point that is the nearest neighbor to the first edge feature point, and search for the second plane feature point that is the nearest neighbor to the first plane feature point.

[0061] Step 14: Based on the relative pose, determine the three-dimensional lidar point cloud that matches the first edge feature point on the prior map and the three-dimensional lidar point cloud that matches the first plane feature point, respectively. The "three-dimensional lidar point cloud that matches the first edge feature point" is referred to as the "prior matching edge point cloud", and the "three-dimensional lidar point cloud that matches the first plane feature point" is referred to as the "prior matching plane point cloud".

[0062] The previously matched edge point cloud can be expressed as f1(A1,O), where A1 represents the pose of the first edge feature point, and O represents the relative pose between the current frame 3D lidar point cloud and the previous map.

[0063] The prior matched plane point cloud can be expressed as f2(A2,O), where A2 represents the pose of the first plane feature point, and O represents the relative pose between the current frame 3D lidar point cloud and the prior map.

[0064] Step 15: Establish an edge feature point loss function based on the previously matched edge point cloud and the second edge feature point, and establish a plane feature point loss function based on the previously matched plane point cloud and the second plane feature point.

[0065] The edge feature point loss function refers to the distance error equation of the line segment formed by the first matching edge point cloud to the second edge feature point. The loss value of the edge feature point loss function is "the distance between the line segment formed by the first matching edge point cloud and the second edge feature point".

[0066] The plane feature point loss function refers to the distance error equation of the plane formed by the first matching plane point cloud to the second plane feature point. The loss value of the plane feature point loss function is "the distance from the first matching plane point cloud to the plane formed by the second plane feature point".

[0067] Step 16: With the loss values of the edge feature point loss function and the plane feature point loss function being less than the preset values as the goal, the relative pose O is iteratively optimized, and the relative pose O with the loss values of the edge feature point loss function and the plane feature point loss function being less than the preset values is taken as the final relative pose.

[0068] Step S107: updating the three-dimensional point cloud map according to the relative posture and the three-dimensional lidar point cloud of the current frame to establish a three-dimensional point cloud map of the driving environment.

[0069] For the first frame of 3D lidar point cloud, there is no prior map, and the first frame of 3D lidar point cloud can be used as the prior map of the second frame of 3D lidar point cloud.

[0070] For each non-first frame 3D lidar point cloud, after determining the relative pose of the current frame 3D lidar point cloud and the previous map, point cloud registration can be performed on each 3D lidar point cloud in the current frame 3D lidar point cloud and the 3D lidar point cloud in the previous map according to this relative pose to determine which 3D lidar point cloud in the previous map each 3D lidar point cloud in the current frame 3D lidar point cloud matches or corresponds to. Finally, according to the result of the point cloud registration, the 3D lidar point cloud in the current frame 3D lidar point cloud is added to the previous map, that is, the 3D lidar point cloud of the current frame is spliced with the previous map to form a new map, which becomes the previous map of the next frame 3D lidar point cloud.

[0071] Based on the method described in steps S101 to S107 above, the global satellite positioning device installed on the vehicle can be reused, and the advantage of the high positioning accuracy of the global satellite positioning device can be utilized. When the positioning signal of the global satellite positioning device is normal, the global satellite positioning device can be used to establish a three-dimensional point cloud map to improve the accuracy of the three-dimensional point cloud map. When the positioning signal of the global satellite positioning device is abnormal, the map is updated based on the relative position of the current frame three-dimensional lidar point cloud and the previous map to establish a three-dimensional point cloud map.

[0072] The above steps S103, S104 and S106 are further described below. First, the above step S103 is described.

[0073] In order to improve the accuracy of the radar positioning posture obtained by coordinate system conversion based on the posture parameters, in some implementations of the above step S103, the posture parameters T for coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle can be determined by the following steps 21 to 23.

[0074] Step 21: Based on the pose parameters (initial pose parameters for coordinate system conversion between the global satellite positioning device and the lidar on the vehicle) and the global positioning pose obtained by the global satellite positioning device when the vehicle turns, determine the radar positioning pose corresponding to each frame of the three-dimensional lidar point cloud collected by the lidar when the vehicle turns.

[0075] Radar positioning pose T o It can be expressed as T o =T go *T g , T go Indicates the pose parameters for coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle, T g Indicates the global positioning pose obtained by the global satellite positioning device.

[0076] Compared with straight driving, the position and posture (posture) of the vehicle will change more significantly when turning. Therefore, the data of the vehicle turning can more accurately determine the posture parameter T for the coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle. go .

[0077] Similar to step S104, based on the above-mentioned posture parameter T go The determined radar positioning pose is not the radar positioning pose actually output by the lidar, but the radar positioning pose obtained by coordinate system conversion that matches the global positioning pose.

[0078] Step 22: Using the point cloud registration algorithm based on the ICP (Iterative Closest Point) algorithm, the pose parameter T is calculated based on the corresponding radar positioning pose of each frame of the 3D lidar point cloud collected by the lidar when the vehicle turns. go Optimize.

[0079] Specifically, a KD-Tree (K-dimensional search tree) of the 3D lidar point cloud is first established; then, based on the KD-Tree, the 3D lidar point clouds with the nearest neighbor matching relationship between two adjacent frames of 3D lidar point clouds are searched; finally, a point cloud registration algorithm based on the ICP (Iterative Closest Point) algorithm is used, and the pose parameters T of the coordinate system transformation between the global satellite positioning device and the lidar on the vehicle are performed based on the 3D lidar point clouds with the nearest neighbor matching relationship. goIn some preferred embodiments, the point cloud registration algorithm based on the ICP algorithm can be a point cloud registration algorithm based on the point-to-plane method in the ICP algorithm. For the sake of brevity, the embodiment of the present invention does not specifically describe the point-to-plane method.

[0080] Step 23: The optimized pose parameters T go As the final pose parameter T go , that is, the pose parameter T used in step S104 go .

[0081] Based on the method described in steps 21 to 23 above, the posture parameter T can be significantly improved. go calibration accuracy.

[0082] The above is a further explanation of step S103.

[0083] The above step S104 will be further described below.

[0084] In order to accurately obtain the radar positioning pose corresponding to each frame of the three-dimensional lidar point cloud, in some implementations of the above step S104, the radar positioning pose can be obtained through the following steps 31 to 32.

[0085] Step 31: According to the timestamp corresponding to each frame of the three-dimensional lidar point cloud, the global positioning pose obtained by the global satellite positioning device is interpolated to obtain the global positioning pose at each timestamp.

[0086] Because the GPS and LiDAR acquisition times may be out of sync, the LiDAR may sometimes capture a new frame of 3D LiDAR point cloud, but the GPS may not capture the new frame of positioning signals until a slight delay. This means that the 3D LiDAR point cloud actually captured by the LiDAR and the global positioning pose of the positioning signal collected by the GPS are not one-to-one corresponding. However, by using the above-mentioned method of timestamp interpolation of the global positioning pose, we can accurately determine the global positioning pose corresponding to each frame of the 3D LiDAR point cloud, effectively obtaining a one-to-one global positioning pose corresponding to each frame of the 3D LiDAR point cloud actually captured by the LiDAR.

[0087] Step 32: Pose parameters T based on the coordinate system conversion between the global satellite positioning device and the lidar go , and according to the global positioning pose at each timestamp, the corresponding radar positioning pose of each frame of 3D lidar point cloud is determined respectively.

[0088] Based on the method described in steps 31 to 32 above, the radar positioning pose corresponding to each frame of three-dimensional lidar point cloud actually collected by the lidar can be accurately obtained.

[0089] The above is a further explanation of step S104.

[0090] The above step S106 will be further described below.

[0091] According to step 15 in the above step S106, when determining the relative pose of the current frame three-dimensional lidar point cloud with respect to the previous map, the geometric structural features of the point cloud (the distance between the line segment formed by the previously matched edge point cloud and the second edge feature point and the distance between the previously matched plane point cloud and the plane formed by the second plane feature point) are mainly aligned to determine the relative pose. However, when the vehicle is traveling in a scene with a strong repetitiveness of geometric structural features (such as a highway, a tunnel or a cross-sea bridge, etc.), the use of the above method of aligning the geometric structural features of the point cloud to determine the relative pose may cause mismatching. In this regard, in order to overcome the above problem and improve the robustness of the method for determining the relative pose, in this embodiment, the relative pose of the current frame three-dimensional lidar point cloud with respect to the previous map can be determined by the following steps 41 to 42.

[0092] Step 41: Based on the angular velocity and linear velocity of the vehicle, predict the relative pose of the current frame 3D lidar point cloud relative to the 3D point cloud map established based on the previous multiple frames of 3D lidar point clouds, and use the predicted relative pose as the prior relative pose.

[0093] Angular velocity refers to the arc traveled per unit time (e.g., 1 second), while linear velocity refers to the distance traveled per unit time (e.g., 1 second). When the GPS positioning signal is abnormal, the angular velocity and linear velocity can be received from the vehicle's wheel odometer via the CAN (Controller Area Network) bus. When the GPS positioning signal is normal, the angular velocity and linear velocity can be obtained from the GPS.

[0094] The angular velocity and linear velocity of the vehicle usually do not change suddenly within one or several scanning cycles (short time) of the lidar. Therefore, the posture change of the current frame 3D lidar point cloud can be predicted based on the angular velocity of the vehicle in the scanning cycle corresponding to the previous frame or the previous few frames of the 3D lidar point cloud. At the same time, the position change of the current frame 3D lidar point cloud can also be predicted based on the linear velocity of the vehicle in the scanning cycle corresponding to the previous frame or the previous few frames of the 3D lidar point cloud. Based on the above posture change and attitude change, the prediction result of the relative posture of the current frame 3D lidar point cloud relative to the previous map can be obtained.

[0095] Step 42: Based on the prior relative pose, determine the relative pose of the current frame 3D LiDAR point cloud relative to the 3D point cloud map established based on the previous multiple frames of 3D LiDAR point clouds. Specifically, first, the method described in steps 11 to 16 in the aforementioned embodiment is used to determine the relative pose of the current frame 3D LiDAR point cloud relative to the previous map. For the sake of simplicity, this relative pose is referred to as the initial relative pose. Then, the initial relative pose is compared with the prior relative pose; if the difference between the two is small, the initial relative pose is used as the final relative pose; if the difference between the two is large, the prior relative pose is used as the final relative pose.

[0096] Based on the method described in steps 41 to 42 above, even when the vehicle is driving in a scene with highly repetitive geometric structure features, the relative position of the current frame three-dimensional lidar point cloud relative to the previous map can be accurately obtained.

[0097] In addition, in other embodiments, in order to further improve the robustness of the method for determining relative pose, before determining the relative pose of the current frame three-dimensional lidar point cloud relative to the previous map, dynamic object detection can be performed on the current frame three-dimensional lidar point cloud, and the three-dimensional lidar point cloud belonging to dynamic objects in the current frame three-dimensional lidar point cloud can be removed according to the detection results. Finally, the relative pose is determined based on the current frame three-dimensional lidar point cloud after removing the three-dimensional lidar point cloud belonging to dynamic objects.

[0098] In this embodiment, 3D lidar point cloud samples can be used to train a dynamic object detection model based on a deep convolutional neural network, and then the trained dynamic object detection model is used to perform dynamic object detection on each frame of 3D lidar point cloud. The dynamic object detection model can output a detection frame for each dynamic object in each frame of 3D lidar point cloud. The 3D lidar point cloud within the detection frame is the 3D lidar point cloud belonging to the dynamic object, and these 3D lidar point clouds can be removed.

[0099] Furthermore, in this embodiment, in order to avoid drift of the radar positioning posture and / or three-dimensional point cloud map corresponding to the current frame three-dimensional lidar point cloud actually output by the lidar, after updating the three-dimensional point cloud map based on the relative posture of the current frame three-dimensional lidar point cloud relative to the previous map and the current frame three-dimensional lidar point cloud to establish a three-dimensional point cloud map of the driving environment, the radar positioning posture corresponding to the current frame three-dimensional lidar point cloud can also be optimized through the following step 51.

[0100] Step 51: According to the constraints of the preset factor graph (Pose Graph), a pose optimization method based on the factor graph (Pose Graph) is used to optimize the radar positioning pose corresponding to the current frame 3D lidar point cloud. It should be noted that the pose optimization method based on the factor graph (Pose Graph) is a conventional pose optimization method in the field of positioning and mapping technology. This embodiment does not explain the specific principles of this method, but only explains the constraints used by this method.

[0101] The constraints of the preset factor graph (Pose Graph) may include radar positioning pose constraints and global positioning pose constraints. The radar positioning pose constraints and global positioning pose constraints are described below.

[0102] (1) Radar positioning posture constraints.

[0103] In this embodiment, the radar positioning posture constraint condition is shown in the following formula (1).

[0104]

[0105] The meanings of the parameters in formula (1) are as follows:

[0106] r odom represents the constraint error of the radar positioning pose constraint condition, represents the absolute pose of the i-1th key frame in the Pose Graph, represents the absolute pose of the i-th key frame in the Pose Graph, Express Perform the inverse operation, represents the relative pose of the i-th keyframe relative to the i-1-th keyframe, w represents the world coordinate system, and b represents the vehicle coordinate system. The absolute pose refers to the pose relative to the world coordinate system.

[0107] A keyframe is a frame of 3D LiDAR point cloud where the change in radar pose compared to the previous frame is greater than a preset threshold. This means that when building the Pose Graph, the Pose Graph is constructed based on those 3D LiDAR point clouds with large changes in radar pose, while those with small changes are removed. Because 3D LiDAR point clouds with small changes in radar pose have a relatively small impact on vehicle positioning and map updates, and these 3D LiDAR point clouds are also relatively numerous, removing these 3D LiDAR point clouds with small changes in radar pose significantly improves the efficiency of pose optimization based on the Pose Graph.

[0108] (2) Global positioning posture constraints.

[0109] In this embodiment, the global positioning posture constraint condition is shown in the following formula (2).

[0110]

[0111] The meanings of the parameters in formula (2) are as follows:

[0112] r rtk represents the constraint error of the global positioning pose constraint, represents the global positioning pose at the timestamp corresponding to the j-th key frame in the Pose Graph, T rb Indicates the pose parameters for the coordinate system conversion between the vehicle coordinate system and the global satellite positioning device. represents the absolute pose of the jth keyframe in the Pose Graph, Express Perform an inverse operation, r represents the coordinate system of the global satellite positioning device, j∈Ω, and Ω represents the key frame in the three-dimensional lidar point cloud collected by the lidar when the positioning signal of the global satellite positioning device is normal.

[0113] The constraint objective of the above-mentioned preset factor graph (Pose Graph) constraint condition is to make the constraint error r odom and constraint error r rtk The sum is minimized.

[0114] The above is Figure 1 Detailed description of step S106 of the method embodiment is shown.

[0115] The following continues to describe an embodiment of a method for acquiring lane marking data.

[0116] 1. First Embodiment of a Method for Acquiring Lane Marking Data

[0117] See attached Figure 2 , Figure 2 FIG. 1 is a flow chart showing the main steps of the method for obtaining lane marking data according to an embodiment of the present invention. Figure 2 As shown, the method for acquiring lane marking data in the embodiment of the present invention mainly includes the following steps S201 to S204.

[0118] Step S201: Acquire a three-dimensional laser radar point cloud of the driving environment collected by a laser radar on the vehicle during the vehicle's driving process.

[0119] Step S202: Create a three-dimensional point cloud map of the driving environment based on the three-dimensional laser radar point cloud.

[0120] In an embodiment of the present invention, the method described in the aforementioned embodiment of the method for establishing a three-dimensional point cloud map can be used to establish a three-dimensional point cloud map of the driving environment based on the three-dimensional laser radar point cloud.

[0121] Step S203: Generate a bird's eye view (Bird Eye View) and a height map (Height Map) of the driving environment according to the three-dimensional point cloud map of the driving environment.

[0122] A bird's-eye view refers to an image obtained by projecting the 3D lidar point cloud in the 3D point cloud map onto a plane perpendicular to the height direction of the point cloud. Each image point in the image corresponds one-to-one to each 3D lidar point cloud in the 3D point cloud map.

[0123] Each image point in the height map also corresponds one-to-one to each 3D lidar point cloud in the 3D point cloud map, and each image point stores the point cloud height of its corresponding 3D lidar point cloud.

[0124] It should be noted that those skilled in the art can adopt the conventional bird's-eye view generation method in the field of three-dimensional lidar point cloud processing technology and generate a bird's-eye view of the driving environment based on the three-dimensional lidar point cloud in the three-dimensional point cloud map. In addition, they can also adopt the conventional height map generation method and generate a height map of the driving environment based on the three-dimensional lidar point cloud in the three-dimensional point cloud map. The embodiments of the present invention do not specifically limit the above methods.

[0125] Step S204: labeling lane elements of the driving environment during vehicle driving according to the bird's-eye view and the height map to form four-dimensional lane element labeling data including one-dimensional time information and three-dimensional spatial position information.

[0126] The 3D point cloud map is created based on the 3D LiDAR point cloud during vehicle travel, so this 3D point cloud map is a dynamic map that contains time information. Furthermore, the bird's-eye view and height map determined based on this 3D point cloud map are also dynamic images that contain time information. Lane elements at least include traffic signs on the lane and / or other objects that can serve as markings, where traffic signs at least include lane lines, stop lines, road signs (such as left-turn arrows), traffic lights, and traffic signs, etc. Other objects that can serve as markings include at least rod-shaped objects.

[0127] When lane element annotation is performed based on the aforementioned bird's-eye view and height map, the lane element annotation is actually performed based on a sequence of bird's-eye view and height map arranged in chronological order, thereby generating four-dimensional lane element annotation data containing one-dimensional time information and three-dimensional spatial position information. The one-dimensional time information can be determined from the time information of the bird's-eye view or height map, the two-dimensional plane position in the three-dimensional spatial position information can be determined from the bird's-eye view, and the height information in the three-dimensional spatial position information can be determined from the height map.

[0128] Based on the method described in steps S201 to S204 above, the annotation data of lane elements in four-dimensional space (a four-dimensional space formed by time and three-dimensional space) can be accurately obtained. After using these annotation data to train the vehicle's perception model, the perception ability of the perception model can be greatly improved, so that the real-time local map and obstacle information of the driving environment can be accurately determined during the vehicle's driving process, ensuring that the vehicle can drive safely.

[0129] So far, the first embodiment of the method for obtaining lane marking data has been described with reference to the accompanying drawings. Now, the second embodiment of the method for obtaining lane marking data will be described.

[0130] 2. Second Embodiment of the Method for Acquiring Lane Marking Data

[0131] In the method for obtaining lane marking data according to the second embodiment of the present invention, the method for obtaining lane marking data may include: Figure 2 Steps S201 to S204 of the method embodiment shown in FIG. Figure 2 The main difference between the embodiment of the method shown is that after executing step S202 and before executing step S203, it also includes a step of removing the 3D LiDAR point cloud belonging to dynamic objects. Figure 3 , Figure 3 The main process of the method for removing the three-dimensional laser radar point cloud belonging to the dynamic object in the embodiment of the present invention is exemplified. Figure 3 As shown, in this embodiment of the present invention, the three-dimensional lidar point cloud belonging to the dynamic object can be removed through the following steps S301 to S302.

[0132] Step S301: performing ground fitting on the 3D lidar point cloud in the 3D point cloud map to determine the ground plane of the 3D point cloud map.

[0133] Step S302: Remove the three-dimensional lidar point clouds belonging to dynamic objects in the three-dimensional point cloud map according to the ground plane. Specifically, the three-dimensional lidar point clouds located above the ground plane can be regarded as three-dimensional lidar point clouds belonging to dynamic objects. In some embodiments, after determining the ground plane, the three-dimensional lidar point cloud can be judged as a ground point cloud or a non-ground point cloud based on the ground height at the position corresponding to each three-dimensional lidar point cloud, and then based on the point cloud height and the ground height corresponding to each three-dimensional lidar point cloud, and finally all non-ground point clouds are removed. Among them, if the point cloud height is less than or equal to the ground height, the three-dimensional lidar point cloud is a ground point cloud; otherwise, the three-dimensional lidar point cloud is a non-ground point cloud.

[0134] Based on the method described in steps S301 to S302 above, the three-dimensional lidar point cloud belonging to dynamic objects in the three-dimensional point cloud map can be removed. In this way, when lane elements are labeled according to the three-dimensional point cloud map, the interference of the three-dimensional lidar point cloud belonging to dynamic objects on the lane elements can be eliminated, thereby improving the labeling efficiency of the lane elements.

[0135] The above step S201 is further explained below.

[0136] See attached Figure 4 In some implementations of the above step S301, the ground plane of the three-dimensional point cloud map can be determined through the following steps S3011 to S3014.

[0137] Step S3011: using a polar coordinate grid representation method based on a concentric zone model, with the center point of the three-dimensional point cloud map as the pole, to establish a polar coordinate grid map of the three-dimensional point cloud map.

[0138] by Figure 5 Taking the polar coordinate grid diagram shown in FIG. 1 as an example, the polar coordinate grid representation method based on the concentric zone model is briefly explained.

[0139] Specifically, with the center point of the three-dimensional point cloud map as the pole, multiple concentric circle areas are formed along the order of the polar diameter values from small to large. The polar diameter values corresponding to each concentric circle area can be the same or different. Figure 5 As shown, four concentric circle areas Z1, Z2, Z3 and Z4 are formed in the order of polar diameter values from small to large, and the polar diameter values corresponding to each of them are different. After determining multiple concentric circle areas, each concentric circle area is gridded again. In the same concentric circle area, the polar angle angle corresponding to each grid (the range of polar angle angles covered by the grid) is the same, and the polar angle angles corresponding to the grids in different concentric circle areas may be different. Figure 5As shown, the polar angles corresponding to the grids in the concentric circle areas Z2 and Z3 are the same and the smallest, followed by the concentric circle area Z1 with the smallest polar angle, and the concentric circle area Z4 with the largest polar angle.

[0140] Step S3012: For each grid in the polar coordinate grid diagram, plane fitting is performed on the three-dimensional lidar point cloud within the grid to obtain multiple planes within the grid.

[0141] In this embodiment, a conventional plane fitting method in the field of plane fitting technology can be used to perform plane fitting on the three-dimensional lidar point cloud in each grid. For example, a RANSAC (RANdom Sample Consensus) algorithm can be used for plane fitting.

[0142] Step S3013: Determine whether the grid is a ground grid based on the plane normal vector of each plane and the height of each three-dimensional lidar point cloud in the grid, and with the constraint conditions that the vertical angle deviation of each plane normal vector is less than a preset angle deviation threshold and the height difference between adjacent three-dimensional lidar point clouds in the grid is less than a preset height difference threshold.

[0143] Vertical angle deviation refers to the angular deviation between the plane normal vector and the vertical direction of the 3D LiDAR point cloud. The vector [0 0 1] can be used to represent vertically upward, or the vector [0 0-1] can be used to represent vertically downward. After determining the plane normal vector, the vector angle between the plane normal vector and the vertically upward vector [0 01] (or vertically downward vector [0 0-1]) can be calculated using the vector angle calculation method. This vector angle is used as the vertical angle deviation.

[0144] If the vertical angle deviation is less than the preset angle deviation threshold, it indicates that the plane normal vector is perpendicular to the ground, and the plane corresponding to the plane normal vector is likely the ground plane. If the height difference between adjacent 3D LiDAR point clouds within a grid is less than the preset height difference threshold, it indicates that the 3D LiDAR point cloud within the grid is relatively smooth, and the grid may also be a ground grid. If both the vertical angle deviation is less than the preset angle deviation threshold and the height difference between adjacent 3D LiDAR point clouds within the grid is less than the preset height difference threshold are met, the grid can be determined to be a ground grid.

[0145] Step S3014: performing ground fitting on the three-dimensional lidar point cloud within the ground grid to determine the ground plane of the three-dimensional point cloud map.

[0146] In this embodiment, a conventional plane fitting method in the field of plane fitting technology can be used to perform ground fitting on the three-dimensional lidar point cloud within the ground grid to determine the ground plane of the three-dimensional point cloud map, which is not described in detail in this embodiment.

[0147] Based on the method described in steps S3011 to S3014 above, the ground plane of the three-dimensional point cloud map can be accurately obtained, thereby improving the accuracy of the ground point cloud and non-ground point cloud, and ultimately improving the accuracy of removing the three-dimensional lidar point cloud belonging to dynamic objects.

[0148] So far, the second embodiment of the method for obtaining lane marking data has been described with reference to the accompanying drawings. Now, the third embodiment of the method for obtaining lane marking data will be described.

[0149] 3. Third Embodiment of the Method for Acquiring Lane Marking Data

[0150] In the method for obtaining lane marking data according to the third embodiment of the present invention, the method for obtaining lane marking data includes: Figure 2 Steps S201 to S204 of the method embodiment shown in FIG. Figure 2 The main difference between the illustrated method embodiment is that, after executing step S202 and before executing step S203, the method further includes a step of coloring the 3D LiDAR point cloud. Specifically, the step of coloring the 3D LiDAR point cloud may include:

[0151] The three-dimensional point cloud map of the driving environment is colored based on the color information of the two-dimensional image of the driving environment captured by the image acquisition device on the vehicle, so that the driving environment can be visualized when generating a bird's eye view of the driving environment.

[0152] Specifically, the color information of the two-dimensional image can be added to the point cloud information of the three-dimensional lidar point cloud in the three-dimensional point cloud map to achieve coloring processing of the three-dimensional point cloud map.

[0153] In some preferred implementations, the three-dimensional point cloud map of the driving environment may be colored through the following steps S401 to S402.

[0154] Step S401: Determine the clear three-dimensional lidar point cloud and the unclear three-dimensional lidar point cloud based on the laser reflection intensity of each three-dimensional lidar point cloud in the three-dimensional point cloud map. Specifically, the laser reflection intensity can be compared with a preset intensity threshold; if the laser reflection intensity is less than the preset intensity threshold, the three-dimensional lidar point cloud is an unclear three-dimensional lidar point cloud; if the laser reflection intensity is greater than or equal to the preset intensity threshold, the three-dimensional lidar point cloud is a clear three-dimensional lidar point cloud. Those skilled in the art can flexibly set the specific value of the preset intensity threshold according to actual needs, and this embodiment does not specifically limit this. In some preferred embodiments, the clear three-dimensional lidar point cloud and the unclear three-dimensional lidar point cloud in each frame of the three-dimensional lidar point cloud of the three-dimensional point cloud map can be determined separately by parallel processing to improve processing efficiency.

[0155] Step S402: The unclear three-dimensional lidar point cloud is colorized according to the color information of the two-dimensional image of the driving environment, so that the unclear three-dimensional lidar point cloud can be visualized when generating a bird's eye view of the driving environment. Specifically, the color information of the two-dimensional image can be added to the point cloud information of the unclear three-dimensional lidar point cloud to achieve the colorization of the three-dimensional point cloud map. Similar to step S401, in some preferred embodiments, the unclear three-dimensional lidar point cloud in each frame of the three-dimensional lidar point cloud can also be colorized according to the color information of the two-dimensional image of the driving environment collected by the image acquisition device on the vehicle in a parallel processing manner to improve processing efficiency.

[0156] Based on the method described in steps S401 to S402 above, the unclear three-dimensional lidar point cloud can be colored to improve the visualization of the unclear three-dimensional lidar point cloud.

[0157] The above step S402 is further explained below.

[0158] In some implementations of the above step S402, the unclear three-dimensional lidar point cloud can be colored through the following steps S4021 to S4022.

[0159] Step S4021: Determine the nearest neighbor two-dimensional image corresponding to each frame of three-dimensional lidar point cloud according to the timestamp corresponding to each frame of three-dimensional lidar point cloud and the timestamp corresponding to each frame of two-dimensional image collected by the image acquisition device.

[0160] Step S4022: Based on the nearest neighbor two-dimensional image, the unclear three-dimensional lidar point cloud in each frame of the three-dimensional lidar point cloud is colorized. Specifically, first, based on the timestamp corresponding to the nearest neighbor two-dimensional image, the radar positioning pose obtained by the lidar is interpolated to obtain the radar positioning pose at the timestamp; then, based on the pose parameters of the coordinate system conversion between the lidar and the image acquisition device, and based on the radar positioning pose at the above timestamp, the image positioning pose corresponding to the nearest neighbor two-dimensional image is determined; then, based on the image positioning pose and the position information of the unclear three-dimensional lidar point cloud in the radar coordinate system, the projected image point after projecting the unclear three-dimensional lidar point cloud onto the nearest neighbor two-dimensional image is determined; finally, the unclear three-dimensional lidar point cloud is colorized based on the color information of this projected image point.

[0161] Furthermore, when there are multiple image acquisition devices on the vehicle and the nearest neighbor two-dimensional image corresponding to the current frame three-dimensional lidar point cloud includes two-dimensional images acquired by multiple image acquisition devices, the two-dimensional image acquired by the last image acquisition device can be selected as the final nearest neighbor two-dimensional image according to the preset arrangement order of the image acquisition devices, and then the unclear three-dimensional lidar point cloud in the current frame three-dimensional lidar point cloud can be colored according to the final nearest neighbor two-dimensional image.

[0162] Assume that a vehicle is equipped with six image acquisition devices (A, B, C, D, E, and F), arranged in the order ABCDEF. For example, the nearest neighbor 2D image corresponding to the current frame's 3D lidar point cloud includes 2D images a, b, c, d, e, and f captured by image acquisition devices A, B, C, D, E, and F. Since image acquisition device F is the last one in the arrangement, 2D image f is used as the final nearest neighbor 2D image.

[0163] For another example, the nearest neighbor two-dimensional image corresponding to the three-dimensional lidar point cloud of the current frame includes the two-dimensional images a, c and e captured by the three image acquisition devices A, C and E. Since the last image acquisition device is E, the two-dimensional image e is used as the final nearest neighbor two-dimensional image.

[0164] Based on the method described in steps S4021 to S4022 above, even when multiple image acquisition devices are installed on a vehicle, unclear three-dimensional lidar point clouds can be accurately and reliably colored.

[0165] In some other implementations of the above step S402, the unclear three-dimensional lidar point cloud can be colored through the following steps S4023 to S4024.

[0166] Step S4023: extracting a preset driving environment region of interest (Region of Interest) from the two-dimensional image.

[0167] The preset driving environment region of interest (ROI) includes at least the lane area in the driving environment and excludes areas that may interfere with point cloud shading. The areas that may interfere with point cloud shading include at least the vehicle image area and areas of interference caused by image exposure. The areas of interference caused by image exposure may be areas of interference caused by the rolling shutter effect.

[0168] Step S4024: Colorize the unclear 3D LiDAR point cloud according to the color information of the preset driving environment region of interest. The implementation of this step is similar to the method described in step S4022 above and will not be repeated here.

[0169] Furthermore, in the method for obtaining lane marking data according to the third embodiment of the present invention, the method for obtaining lane marking data includes: Figure 2 Step S203 and step S204 of the illustrated method embodiment may be replaced by the following steps S403 to S405 , respectively.

[0170] Step S403: Generate a first bird's eye view (BEV) based on the clear 3D LiDAR point cloud in the colored 3D point cloud map, generate a second BEV based on the unclear 3D LiDAR point cloud in the colored 3D point cloud map, and generate a height map based on the colored 3D point cloud map. In the first BEV, each image point corresponds one-to-one with the clear 3D LiDAR point cloud; in the second BEV, each image point corresponds one-to-one with the unclear 3D LiDAR point cloud.

[0171] Step S404: In response to receiving the marking start instruction, loading and displaying a first bird's eye view and a height map.

[0172] Step S405: During the process of loading the first bird's eye view and the height map, if a label switching instruction is received, the second bird's eye view and the height map are loaded and displayed.

[0173] Based on the method described in steps S403 to S405 above, the first bird's eye view can be loaded first when starting the lane element labeling work. When certain lane elements cannot be clearly seen from the first bird's eye view, the second bird's eye view can be switched to load to continue completing the lane element labeling work.

[0174] Furthermore, in the method for obtaining lane marking data according to the third embodiment of the present invention, the method for obtaining lane marking data may include: Figure 2 Steps S201 to S204 of the method embodiment shown may also include Figure 3 Steps S301 to S302 of the method embodiment shown (the method for obtaining lane marking data of the second embodiment) At this time, the step of coloring the 3D lidar point cloud can be performed after executing step S302 and before executing step S203.

[0175] So far, the third embodiment of the method for obtaining lane marking data has been described with reference to the accompanying drawings. Now, the fourth embodiment of the method for obtaining lane marking data will be described.

[0176] 4. Fourth Embodiment of the Method for Acquiring Lane Marking Data

[0177] In the method for obtaining lane marking data according to the fourth embodiment of the present invention, the method for obtaining lane marking data may include: Figure 2 Steps S201 to S204 of the method embodiment shown in FIG. Figure 2 The main difference between the embodiment of the method shown is that after executing step S204, it also includes a step of calibrating the posture parameters of the laser radar and the image acquisition device for coordinate system conversion. Figure 6 , Figure 6 The main process of the method for calibrating the pose parameters of the laser radar and the image acquisition device for coordinate system conversion in an embodiment of the present invention is exemplified. Figure 6 As shown, in an embodiment of the present invention, the posture parameters of the laser radar and the image acquisition device for coordinate system conversion can be calibrated through the following steps S501 to S502.

[0178] Step S501: Acquire two-dimensional lane element detection data obtained by performing lane element detection on a two-dimensional image of a driving environment captured by an image capture device on a vehicle.

[0179] In this embodiment, two-dimensional image samples can be used to train a two-dimensional lane element detection model based on a deep convolutional neural network, and then the trained two-dimensional lane element detection model is used to perform lane element detection on the two-dimensional image.

[0180] Step S502: Calibrate the pose parameters of the laser radar and the image acquisition device for coordinate system conversion based on the four-dimensional lane element annotation data and the two-dimensional lane element detection data.

[0181] See attached Figure 7 In this embodiment, the pose parameters of the laser radar and the image acquisition device for coordinate system conversion can be calibrated through the following steps S5021 to S5024.

[0182] Step S5021: Project the four-dimensional lane element annotation data to the coordinate system of the image acquisition device according to the posture parameters to obtain the projection point f(A, O) of the four-dimensional lane element annotation data.

[0183] Where A represents the coordinates of the four-dimensional lane element annotation data in the laser radar coordinate system, and O represents the pose parameters.

[0184] Step S5022: Determine the two image points on the two-dimensional image that are closest to the projection point f(A, O).

[0185] Step S5023: Based on the line segment cd formed by the projection point f(A, O) and the two image points, establish a distance error equation from the projection point f(A, O) to the line segment cd.

[0186] The distance error equation from the projection point f(A,O) to the line segment cd is shown in equation (3).

[0187] loss=d(f(A,O),cd) (3)

[0188] The meanings of the parameters in the above formula (3) are as follows:

[0189] d represents the distance calculation function from the projection point f(A,O) to the line segment cd, and loss represents the distance calculated by the distance calculation function d.

[0190] Step S5024: With the goal of making the distance loss less than the preset distance threshold, iteratively optimize the posture parameter O in the distance error equation, and obtain the posture parameter O when the distance loss is less than the preset distance threshold, and use the posture parameter O as the calibrated posture parameter.

[0191] In the embodiment of the present invention, a least squares algorithm can be used to iteratively optimize the pose parameter O in the distance error equation. For example, the Levenberg-Marquardt algorithm in the least squares algorithm can be used to iteratively optimize the pose parameter O in the distance error equation. The embodiment of the present invention does not specifically limit the above iterative optimization method.

[0192] Based on the method described in steps S5021 to S5024 above, the accuracy of calibrating the posture parameters of the coordinate system conversion between the laser radar and the image acquisition device can be improved.

[0193] Furthermore, in the lane marking data acquisition method according to the fourth embodiment of the present invention, in addition to including the aforementioned steps S501 to S502, the lane marking data acquisition method can also determine the lane element marking data on the two-dimensional image of the driving environment based on the calibrated posture parameters and the four-dimensional lane element marking data. Specifically, the four-dimensional lane element marking data can be subjected to a coordinate system transformation based on the calibrated posture parameters to determine the image points corresponding to the four-dimensional lane element marking data on the two-dimensional image, and the lane element marking data on the two-dimensional image can be determined based on these image points. In this way, the efficiency and accuracy of obtaining the lane element marking data on the two-dimensional image can be greatly improved.

[0194] The above is a description of the fourth embodiment of the method for acquiring lane marking data.

[0195] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0196] Those skilled in the art will appreciate that all or part of the processes in the method for implementing the above-mentioned embodiment of the present invention may also be accomplished by instructing the relevant hardware through a computer program. The computer program may be stored in a computer-readable storage medium. When the computer program is executed by a processor, it may implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal, and software distribution medium capable of carrying the computer program code. It should be noted that the content contained in the computer-readable storage medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0197] Furthermore, the present invention also provides a computer device. In an embodiment of a computer device according to the present invention, the computer device includes a processor and a storage device. The storage device can be configured to store a program for executing the method for establishing a three-dimensional point cloud map or the method for obtaining lane marking data of the above-mentioned method embodiment. The processor can be configured to execute the program in the storage device, which includes but is not limited to a program for executing the method for establishing a three-dimensional point cloud map or the method for obtaining lane marking data of the above-mentioned method embodiment. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer device can be a control device device formed by various electronic devices.

[0198] Furthermore, the present invention also provides a computer-readable storage medium. In an embodiment of a computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program for executing the method for establishing a three-dimensional point cloud map or the method for obtaining lane marking data of the above-mentioned method embodiment. The program can be loaded and run by the processor to implement the above-mentioned method for establishing a three-dimensional point cloud map or the method for obtaining lane marking data. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-temporary computer-readable storage medium.

[0199] Thus far, the technical solution of the present invention has been described in conjunction with an embodiment shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A method for obtaining lane marking data, characterized in that: The method comprises: Acquire a three-dimensional laser radar point cloud of the driving environment collected by the laser radar on the vehicle during the vehicle's driving process; Creating a three-dimensional point cloud map of the driving environment based on the three-dimensional laser radar point cloud; generating a bird's-eye view and a height map of the driving environment respectively according to the three-dimensional point cloud map of the driving environment; Annotating lane elements of the driving environment during vehicle driving according to the bird's-eye view and the height map to form four-dimensional lane element annotation data including one-dimensional time information and three-dimensional spatial position information; The step of establishing a three-dimensional point cloud map of the driving environment based on the three-dimensional laser radar point cloud comprises: Acquiring a positioning signal from a global satellite positioning device on the vehicle while the vehicle is traveling and determining whether the positioning signal is normal; If normal, then determining the radar positioning pose corresponding to each frame of the three-dimensional lidar point cloud of the driving environment collected by the lidar based on the pose parameters of the coordinate system conversion between the global satellite positioning device and the lidar on the vehicle and according to the global positioning pose obtained by the global satellite positioning device, and splicing each frame of the three-dimensional lidar point cloud according to the radar positioning pose to establish a three-dimensional point cloud map of the driving environment; If it is abnormal, for each frame of three-dimensional lidar point cloud, determine the relative posture of the current frame three-dimensional lidar point cloud relative to the three-dimensional point cloud map established based on the previous multiple frames of three-dimensional lidar point cloud, and update the three-dimensional point cloud map based on the relative posture and the current frame three-dimensional lidar point cloud to establish a three-dimensional point cloud map of the driving environment.

2. The method for obtaining lane marking data according to claim 1, characterized in that: The step of "determining, based on the pose parameters obtained by coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle and according to the global positioning pose obtained by the global satellite positioning device, the radar positioning pose corresponding to each frame of the three-dimensional laser radar point cloud of the driving environment collected by the laser radar" specifically includes: According to the timestamps corresponding to each frame of the three-dimensional laser radar point cloud, interpolation calculations are performed on the global positioning poses obtained by the global satellite positioning device to obtain the global positioning poses at each timestamp; Based on the pose parameters of the coordinate system conversion between the global satellite positioning device and the laser radar, and according to the global positioning pose at each time stamp, the radar positioning pose corresponding to each frame of the three-dimensional laser radar point cloud is determined respectively.

3. The method for obtaining lane marking data according to claim 1, characterized in that: Prior to the step of "determining, based on the pose parameters of the coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle and according to the global positioning pose obtained by the global satellite positioning device, the radar positioning pose corresponding to each frame of the three-dimensional laser radar point cloud of the driving environment collected by the laser radar", the method further includes optimizing the pose parameters by the following means: Determining, based on the posture parameters and the global positioning posture obtained by the global satellite positioning device when the vehicle turns, the radar positioning posture corresponding to each frame of the three-dimensional lidar point cloud collected by the lidar when the vehicle turns; A point cloud registration algorithm based on the ICP algorithm is adopted to optimize the posture parameters according to the radar positioning posture corresponding to each frame of the three-dimensional lidar point cloud collected by the lidar when the vehicle turns.

4. The method for obtaining lane marking data according to claim 1, characterized in that: The steps of "determining the relative pose of the current frame 3D lidar point cloud relative to the 3D point cloud map established based on the previous multiple frames of 3D lidar point clouds" specifically include: Predicting, based on the angular velocity and linear velocity of the vehicle, a relative pose of the current frame 3D lidar point cloud relative to the 3D point cloud map established based on the previous multiple frames of 3D lidar point clouds, and using the predicted relative pose as a priori relative pose; Based on the prior relative pose, the relative pose of the current frame three-dimensional lidar point cloud relative to the three-dimensional point cloud map established based on the previous multiple frames of three-dimensional lidar point clouds is determined.

5. The method for obtaining lane marking data according to claim 1, characterized in that: After the step of “updating the three-dimensional point cloud map according to the relative pose and the current frame three-dimensional lidar point cloud to establish a three-dimensional point cloud map of the driving environment”, the method further includes: According to the constraints of the preset factor graph, a factor graph-based pose optimization method is used to optimize the radar positioning pose corresponding to the three-dimensional lidar point cloud of the current frame; The constraints of the preset factor graph include radar positioning pose constraints and global positioning pose constraints; The radar positioning posture constraint condition is as follows: Among them, r odom represents the constraint error of the radar positioning pose constraint condition, represents the absolute pose of the i-1th keyframe in the factor graph, represents the absolute pose of the i-th keyframe in the factor graph, Express Perform the inverse operation, represents the relative pose of the i-th keyframe relative to the i-1-th keyframe, w represents the world coordinate system, b represents the vehicle coordinate system, and the keyframe is a frame of 3D LiDAR point cloud whose radar positioning pose changes by more than a preset threshold compared to the previous frame of 3D LiDAR point cloud; The global positioning pose constraint condition is as follows: Among them, r rtk represents the constraint error of the global positioning pose constraint, represents the global positioning pose at the timestamp corresponding to the jth key frame in the factor graph, T rb Indicates the pose parameters for the coordinate system conversion between the vehicle coordinate system and the global satellite positioning device. represents the absolute pose of the jth keyframe in the factor graph, Express An inverse operation is performed, where r represents the coordinate system of the global satellite positioning device, j∈Ω, and Ω represents the key frame in the three-dimensional lidar point cloud collected by the lidar when the positioning signal of the global satellite positioning device is normal.

6. The method for obtaining lane marking data according to claim 1, characterized in that: Before the step of "stitching each frame of the three-dimensional laser radar point cloud according to the radar positioning pose to establish a three-dimensional point cloud map of the driving environment", the method further includes: Each frame of 3D lidar point cloud is subjected to point cloud dedistortion processing.

7. The method for acquiring lane marking data according to claim 1, characterized in that: The steps of "determining the relative pose of the current frame 3D lidar point cloud relative to the 3D point cloud map established based on the previous multiple frames of 3D lidar point clouds" specifically include: Performing dynamic object detection on the three-dimensional laser radar point cloud of the current frame, and removing three-dimensional laser radar point clouds belonging to dynamic objects from the three-dimensional laser radar point cloud of the current frame according to the detection result; The relative posture is determined based on the three-dimensional lidar point cloud of the current frame after removing the three-dimensional lidar point cloud belonging to the dynamic object.

8. A computer device comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, wherein: The program code is suitable for being loaded and run by the processor to execute the method for acquiring lane marking data according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the method for acquiring lane marking data according to any one of claims 1 to 7.

Images