Lane marking data acquisition method, computer equipment and storage medium

Abstract

The present invention relates to the field of autonomous driving technology, and specifically provides a method for obtaining lane annotation data, a computer device, and a storage medium, aiming to solve the problem of improving the accuracy of lane annotation data. To this end, the method provided by the present invention includes establishing a three-dimensional point cloud map based on a three-dimensional lidar point cloud of the driving environment during vehicle driving, generating a bird's-eye view and a height map of the driving environment according to the three-dimensional point cloud map, and finally annotating the lane elements of the driving environment during vehicle driving according to the bird's-eye view and the height map to form four-dimensional lane element annotation data containing one-dimensional time information and three-dimensional spatial position information. Through the above-mentioned implementation, the lane element annotation data in the four-dimensional space formed in time and three-dimensional space can be accurately obtained. After the vehicle's perception model is trained using these annotation data, the perception ability of the perception model can be greatly improved, thereby ensuring that the vehicle can drive safely.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for acquiring lane marking data, a computer device, and a storage medium. Background Art

[0002] Autonomous driving technology mainly includes three key technologies: perception, planning, and control. Among them, perception technology is mainly used to determine the real-time local map and obstacle information of the vehicle's driving environment. Planning technology is mainly used to plan the vehicle's driving trajectory based on the above real-time local map and obstacle information. Control technology is mainly used to control the vehicle to drive according to the planned driving trajectory.

[0003] To improve the accuracy of real-time local maps and obstacle information to ensure safe vehicle driving, the current approach is to first use lane annotation data from lane scenarios to train a multi-sensor perception model based on the BEV (Bird Eye View) perspective. The trained perception model is then used to determine the real-time local map and obstacle information of the vehicle's driving environment. The accuracy of the lane annotation data will greatly affect the accuracy of the perception model, and thus the accuracy of the real-time local map and obstacle information. Therefore, in order to improve the accuracy of lane annotation data, improve the accuracy of the real-time local map and obstacle information, and thus ensure safe vehicle driving, it is necessary to accurately obtain lane annotation data for lane scenarios.

[0004] Accordingly, this field requires a new technical solution to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned defects, the present invention is proposed to provide a method for obtaining lane marking data, a computer device and a storage medium that solves or at least partially solves the technical problem of how to improve the accuracy of lane marking data, so as to improve the accuracy of real-time local maps and obstacle information, and thereby ensure that vehicles can travel safely.

[0006] In a first aspect, a method for acquiring lane marking data is provided, the method comprising:

[0007] Acquire a three-dimensional laser radar point cloud of the driving environment collected by the laser radar on the vehicle during the vehicle's driving process;

[0008] Creating a three-dimensional point cloud map of the driving environment based on the three-dimensional laser radar point cloud;

[0009] Generating a bird's eye view and a height map of the driving environment according to the three-dimensional point cloud map of the driving environment;

[0010] Lane elements of the driving environment during vehicle driving are annotated according to the bird's eye view and the height map to form four-dimensional lane element annotation data including one-dimensional time information and three-dimensional spatial position information.

[0011] In one technical solution of the above-mentioned method for acquiring lane marking data, before the step of "generating a bird's eye view and a height map of the driving environment based on the three-dimensional point cloud map of the driving environment", the method further includes:

[0012] performing ground fitting on the three-dimensional lidar point cloud in the three-dimensional point cloud map to determine a ground plane of the three-dimensional point cloud map;

[0013] The three-dimensional lidar point cloud belonging to a dynamic object in the three-dimensional point cloud map is removed according to the ground plane.

[0014] In one technical solution of the above-mentioned method for acquiring lane annotation data, the step of "performing ground fitting on the three-dimensional lidar point cloud in the three-dimensional point cloud map to determine the ground plane of the three-dimensional point cloud map" specifically includes:

[0015] A polar coordinate grid representation method based on a concentric zone model is adopted, with the center point of the three-dimensional point cloud map as the pole, to establish a polar coordinate grid map of the three-dimensional point cloud map;

[0016] For each grid in the polar coordinate grid diagram, performing plane fitting on the three-dimensional lidar point cloud within the grid to obtain multiple planes within the grid;

[0017] Determining whether the grid is a ground grid based on the plane normal vector of each plane and the height of each three-dimensional lidar point cloud in the grid, and subjecting to the constraints that the vertical angle deviation of each plane normal vector is less than a preset angle deviation threshold and the height difference between adjacent three-dimensional lidar point clouds in the grid is less than a preset height difference threshold;

[0018] Ground fitting is performed on the three-dimensional lidar point cloud within the ground grid to determine the ground plane of the three-dimensional point cloud map.

[0019] In one technical solution of the above-mentioned method for acquiring lane marking data, before the step of "generating a bird's eye view and a height map of the driving environment based on the three-dimensional point cloud map of the driving environment", the method further includes:

[0020] The three-dimensional point cloud map of the driving environment is colored according to the color information of the two-dimensional image of the driving environment captured by the image acquisition device on the vehicle, so that the driving environment can be visualized when generating a bird's-eye view of the driving environment.

[0021] In one technical solution of the above-mentioned method for acquiring lane marking data, the step of "coloring the three-dimensional point cloud map of the driving environment based on the color information of the two-dimensional image of the driving environment captured by the image acquisition device on the vehicle, so as to enable a visual display of the driving environment when generating a bird's-eye view of the driving environment" specifically includes:

[0022] Determining a clear 3D lidar point cloud and an unclear 3D lidar point cloud according to the laser reflection intensity of each 3D lidar point cloud in the 3D point cloud map;

[0023] The unclear three-dimensional lidar point cloud is colored according to the color information of the two-dimensional image of the driving environment so that the unclear three-dimensional lidar point cloud can be visualized when generating a bird's eye view of the driving environment.

[0024] In one technical solution of the above-mentioned method for acquiring lane annotation data, after the step of “annotating lane elements of the driving environment during vehicle driving according to the bird's-eye view and the height map to form four-dimensional lane element annotation data including one-dimensional time information and three-dimensional spatial position information,” the method further includes:

[0025] Acquiring two-dimensional lane element detection data obtained by performing lane element detection on a two-dimensional image of the driving environment captured by an image capture device on the vehicle;

[0026] Calibrate the pose parameters of the laser radar and the image acquisition device for coordinate system conversion based on the four-dimensional lane element annotation data and the two-dimensional lane element detection data;

[0027] and / or,

[0028] The step of “establishing a three-dimensional point cloud map of the driving environment based on the three-dimensional laser radar point cloud” specifically includes:

[0029] For each frame of three-dimensional lidar point cloud, determine the relative pose of the current frame three-dimensional lidar point cloud relative to the three-dimensional point cloud map established based on the previous multiple frames of three-dimensional lidar point clouds, and update the three-dimensional point cloud map based on the relative pose and the current frame three-dimensional lidar point cloud to establish a three-dimensional point cloud map of the driving environment.

[0030] In one technical solution of the above-mentioned method for acquiring lane annotation data, the step of "calibrating the pose parameters for coordinate system conversion of the laser radar and the image acquisition device based on the four-dimensional lane element annotation data and the two-dimensional lane element detection data" specifically includes:

[0031] Projecting the four-dimensional lane element annotation data onto the coordinate system of the image acquisition device according to the pose parameters to obtain a projection point f(A, O) of the four-dimensional lane element annotation data, where A represents the coordinates of the four-dimensional lane element annotation data in the coordinate system of the lidar, and O represents the pose parameters;

[0032] Determine two image points on the two-dimensional image that are closest to the projection point f(A, O);

[0033] According to the line segment cd formed by the projection point f(A, O) and the two image points, the distance error equation from the projection point f(A, O) to the line segment cd is established as shown in the following formula:

[0034] loss=d(f(A,O),cd)

[0035] Wherein, d represents the distance calculation function from the projection point f(A,O) to the line segment cd, and loss represents the distance calculated by the distance calculation function d;

[0036] With the goal of making the distance loss less than a preset distance threshold, the posture parameter O in the distance error equation is iteratively optimized, and the posture parameter O when the distance loss is less than the preset distance threshold is obtained, and the posture parameter O is used as the calibrated posture parameter.

[0037] In one technical solution of the above-mentioned method for acquiring lane annotation data, after the step of “calibrating the pose parameters for coordinate system conversion of the laser radar and the image acquisition device based on the four-dimensional lane element annotation data and the two-dimensional lane element detection data,” the method further includes:

[0038] Based on the calibrated pose parameters and according to the four-dimensional lane element annotation data, lane element annotation data on the two-dimensional image of the driving environment is determined.

[0039] In a second aspect, a computer device is provided, which includes a processor and a storage device, wherein the storage device is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the lane marking data acquisition method described in any one of the technical solutions of the above-mentioned lane marking data acquisition method.

[0040] In a third aspect, a computer-readable storage medium is provided, which stores a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the lane marking data acquisition method described in any one of the technical solutions of the above-mentioned lane marking data acquisition method.

[0041] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects:

[0042] In the technical solution of the present invention, a three-dimensional laser radar point cloud of the driving environment collected by the laser radar on the vehicle during the driving process can be obtained, and then a three-dimensional point cloud map of the driving environment can be established based on the three-dimensional laser radar point cloud; a bird's eye view and a height map of the driving environment are generated respectively according to the three-dimensional point cloud map of the driving environment, and finally, the lane elements of the driving environment during the driving process of the vehicle are annotated according to the bird's eye view and the height map to form four-dimensional lane element annotation data containing one-dimensional time information and three-dimensional spatial position information. Through the above implementation, the annotation data of the lane elements in the four-dimensional space (a four-dimensional space formed by time and three-dimensional space) can be accurately obtained. After the vehicle's perception model is trained using these annotation data, the perception ability of the perception model can be greatly improved, so that the real-time local map and obstacle information of the driving environment can be accurately determined during the vehicle's driving process, ensuring that the vehicle can drive safely.

[0043] Furthermore, in some technical solutions for implementing the present invention, in order to ensure the accuracy of the three-dimensional point cloud map, when establishing the three-dimensional point cloud map, for each frame of three-dimensional lidar point cloud, the relative posture of the current frame three-dimensional lidar point cloud relative to the three-dimensional point cloud map established based on the previous multiple frames of three-dimensional lidar point clouds can be determined, and the three-dimensional point cloud map is updated according to the relative posture and the current frame three-dimensional lidar point cloud to establish a three-dimensional point cloud map of the driving environment.

[0044] Furthermore, in some other technical solutions for implementing the present invention, in order to improve the efficiency of labeling lane elements, the three-dimensional lidar point clouds belonging to dynamic objects in the three-dimensional point cloud map can be removed before generating the bird's-eye view and height map to avoid interference between these three-dimensional lidar point clouds belonging to dynamic objects and the three-dimensional lidar point clouds belonging to lane elements.

[0045] Furthermore, in some other technical solutions for implementing the present invention, in order to further improve the efficiency of labeling lane elements, the three-dimensional point cloud map of the driving environment can be colored according to the color information of the two-dimensional image of the driving environment captured by the image acquisition device on the vehicle, so that the driving environment can be visualized when generating a bird's-eye view of the driving environment, that is, the visualization of the bird's-eye view is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. Those skilled in the art will readily appreciate that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Among them:

[0047] Figure 1 This is a flow chart showing the main steps of a method for acquiring lane marking data according to one embodiment of the present invention;

[0048] Figure 2 This is a flow chart of main steps of a method for establishing a three-dimensional point cloud map when the positioning signal of a global satellite positioning device is normal according to one embodiment of the present invention;

[0049] Figure 3 1 is a flow chart showing the main steps of a method for removing a 3D lidar point cloud belonging to a dynamic object according to an embodiment of the present invention;

[0050] Figure 4 1 is a flow chart showing the main steps of a method for determining a ground plane of a three-dimensional point cloud map according to an embodiment of the present invention;

[0051] Figure 5 is a schematic diagram of a polar coordinate grid diagram of a three-dimensional point cloud map according to one embodiment of the present invention;

[0052] Figure 6 1 is a flow chart of the main steps of a method for calibrating pose parameters for coordinate system conversion between a laser radar and an image acquisition device according to an embodiment of the present invention;

[0053] Figure 7 It is a flowchart of the main steps of a method for calibrating posture parameters for coordinate system conversion between a laser radar and an image acquisition device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0054] Some embodiments of the present invention are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0055] In the description of the present invention, "processor" may include hardware, software, or a combination of the two. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware, or a combination of the two. Computer-readable storage media include any suitable medium that can store program code, such as a magnetic disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, etc. The term "A and / or B" represents all possible combinations of A and B, such as only A, only B, or A and B.

[0056] First, an embodiment of a method for acquiring lane marking data is described below.

[0057] 1. First Embodiment of a Method for Acquiring Lane Marking Data

[0058] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart showing the main steps of the method for obtaining lane marking data according to an embodiment of the present invention. Figure 1 As shown, the method for acquiring lane marking data in the embodiment of the present invention mainly includes the following steps S101 to S104.

[0059] Step S101: Acquire a three-dimensional laser radar point cloud of the driving environment collected by a laser radar on the vehicle during the vehicle's driving process.

[0060] A 3D LiDAR point cloud is three-dimensional data determined by the echo signal reflected back to the LiDAR after receiving the electromagnetic waves sent by the vehicle's LiDAR. This 3D data contains the 3D coordinates of the environmental point in the point cloud coordinate system.

[0061] Step S102: Create a three-dimensional point cloud map of the driving environment based on the three-dimensional laser radar point cloud.

[0062] While the vehicle is driving, the lidar will continuously scan the driving environment to collect three-dimensional lidar point clouds. By splicing each frame of three-dimensional lidar point clouds collected by the lidar in the order of collection, a three-dimensional point cloud map can be obtained.

[0063] Step S103: Generate a bird's eye view and a height map of the driving environment according to the three-dimensional point cloud map of the driving environment.

[0064] A bird's-eye view refers to an image obtained by projecting the 3D lidar point cloud in the 3D point cloud map onto a plane perpendicular to the height direction of the point cloud. Each image point in the image corresponds one-to-one to each 3D lidar point cloud in the 3D point cloud map.

[0065] Each image point in the height map also corresponds one-to-one to each 3D lidar point cloud in the 3D point cloud map, and each image point stores the point cloud height of its corresponding 3D lidar point cloud.

[0066] It should be noted that those skilled in the art can adopt the conventional bird's-eye view generation method in the field of three-dimensional lidar point cloud processing technology and generate a bird's-eye view of the driving environment based on the three-dimensional lidar point cloud in the three-dimensional point cloud map. In addition, they can also adopt the conventional height map generation method and generate a height map of the driving environment based on the three-dimensional lidar point cloud in the three-dimensional point cloud map. The embodiments of the present invention do not specifically limit the above methods.

[0067] Step S104: labeling lane elements of the driving environment during vehicle driving according to the bird's-eye view and the height map to form four-dimensional lane element labeling data including one-dimensional time information and three-dimensional spatial position information.

[0068] The 3D point cloud map is created based on the 3D LiDAR point cloud during vehicle travel, so this 3D point cloud map is a dynamic map that contains time information. Furthermore, the bird's-eye view and height map determined based on this 3D point cloud map are also dynamic images that contain time information. Lane elements at least include traffic signs on the lane and / or other objects that can serve as markings, where traffic signs at least include lane lines, stop lines, road signs (such as left-turn arrows), traffic lights, and traffic signs, etc. Other objects that can serve as markings include at least rod-shaped objects.

[0069] When lane element annotation is performed based on the aforementioned bird's-eye view and height map, the lane element annotation is actually performed based on a sequence of bird's-eye view and height map arranged in chronological order, thereby generating four-dimensional lane element annotation data containing one-dimensional time information and three-dimensional spatial position information. The one-dimensional time information can be determined from the time information of the bird's-eye view or height map, the two-dimensional plane position in the three-dimensional spatial position information can be determined from the bird's-eye view, and the height information in the three-dimensional spatial position information can be determined from the height map.

[0070] Based on the method described in steps S101 to S104 above, the annotation data of lane elements in four-dimensional space (a four-dimensional space formed by time and three-dimensional space) can be accurately obtained. After using these annotation data to train the vehicle's perception model, the perception ability of the perception model can be greatly improved, so that the real-time local map and obstacle information of the driving environment can be accurately determined during the vehicle's driving process, ensuring that the vehicle can drive safely.

[0071] The above step S102 is further explained below.

[0072] To ensure the accuracy of the 3D point cloud map, in some embodiments of step S102 above, when establishing the 3D point cloud map, the relative pose (ego-motion) of the current frame's 3D LiDAR point cloud relative to the 3D point cloud map established based on multiple frames of 3D LiDAR point clouds can be determined for each frame of the 3D LiDAR point cloud. Based on the relative pose and the current frame's 3D LiDAR point cloud, the 3D point cloud map is updated to establish a 3D point cloud map of the driving environment. For simplicity, the "3D point cloud map established based on multiple frames of 3D LiDAR point clouds" will be referred to as the "prior map."

[0073] For the first frame of 3D lidar point cloud, there is no prior map, and the first frame of 3D lidar point cloud can be used as the prior map of the second frame of 3D lidar point cloud.

[0074] For each non-first frame 3D lidar point cloud, after determining the relative pose of the current frame 3D lidar point cloud and the previous map, point cloud registration can be performed on each 3D lidar point cloud in the current frame 3D lidar point cloud and the 3D lidar point cloud in the previous map according to this relative pose to determine which 3D lidar point cloud in the previous map each 3D lidar point cloud in the current frame 3D lidar point cloud matches or corresponds to. Finally, according to the result of the point cloud registration, the 3D lidar point cloud in the current frame 3D lidar point cloud is added to the previous map, that is, the 3D lidar point cloud of the current frame is spliced ​​with the previous map to form a new map, which becomes the previous map of the next frame 3D lidar point cloud.

[0075] Specifically, in this embodiment, the relative position of the current frame three-dimensional lidar point cloud and the previous map can be determined through the following steps 11 to 16.

[0076] Step 11: Filter out the first edge feature point and the first plane feature point from all three-dimensional lidar point clouds contained in the current frame three-dimensional lidar point cloud.

[0077] Specifically, the roughness of each three-dimensional lidar point cloud can be calculated, and the three-dimensional lidar point cloud with a roughness less than a preset roughness threshold can be used as the first plane feature point, and the three-dimensional lidar point cloud with a roughness greater than or equal to the preset roughness threshold can be used as the first edge feature point.

[0078] It should be noted that in this embodiment, the roughness of each 3D lidar point cloud can be calculated using conventional point cloud roughness calculation methods in the field of 3D point cloud processing technology, and this embodiment does not specifically limit this calculation method. In addition, those skilled in the art can flexibly set the specific value of the preset roughness threshold according to actual needs, and this embodiment also does not specifically limit this.

[0079] Step 12: Filter out the second edge feature points and the second plane feature points from the three-dimensional lidar point cloud contained in the previous map, and establish a KD-Tree (K-dimensional search tree) of the three-dimensional lidar point cloud based on the filtered second edge feature points and the second plane feature points.

[0080] The method for selecting the second edge feature points and the second plane feature points from the 3D LiDAR point cloud contained in the prior map is the same as the method for selecting the first edge feature points and the first plane feature points in step 11 above, and will not be repeated here. In addition, in this embodiment, a conventional KD-Tree establishment method can be used to establish a KD-Tree for the 3D LiDAR point cloud based on the selected second edge feature points and the second plane feature points.

[0081] Step 13: Based on the KD-Tree of the 3D lidar point cloud, search for the second edge feature point that is the nearest neighbor to the first edge feature point, and search for the second plane feature point that is the nearest neighbor to the first plane feature point.

[0082] Step 14: Based on the relative pose, determine the three-dimensional lidar point cloud that matches the first edge feature point on the prior map and the three-dimensional lidar point cloud that matches the first plane feature point, respectively. The "three-dimensional lidar point cloud that matches the first edge feature point" is referred to as the "prior matching edge point cloud", and the "three-dimensional lidar point cloud that matches the first plane feature point" is referred to as the "prior matching plane point cloud".

[0083] The previously matched edge point cloud can be expressed as f1(A1,O), where A1 represents the pose of the first edge feature point, and O represents the relative pose between the current frame 3D lidar point cloud and the previous map.

[0084] The prior matched plane point cloud can be expressed as f2(A2,O), where A2 represents the pose of the first plane feature point, and O represents the relative pose between the current frame 3D lidar point cloud and the prior map.

[0085] Step 15: Establish an edge feature point loss function based on the previously matched edge point cloud and the second edge feature point, and establish a plane feature point loss function based on the previously matched plane point cloud and the second plane feature point.

[0086] The edge feature point loss function refers to the distance error equation of the line segment formed by the first matching edge point cloud to the second edge feature point. The loss value of the edge feature point loss function is "the distance between the line segment formed by the first matching edge point cloud and the second edge feature point".

[0087] The plane feature point loss function refers to the distance error equation of the plane formed by the first matching plane point cloud to the second plane feature point. The loss value of the plane feature point loss function is "the distance from the first matching plane point cloud to the plane formed by the second plane feature point".

[0088] Step 16: With the loss values ​​of the edge feature point loss function and the plane feature point loss function being less than the preset values ​​as the goal, the relative pose O is iteratively optimized, and the relative pose O with the loss values ​​of the edge feature point loss function and the plane feature point loss function being less than the preset values ​​is taken as the final relative pose.

[0089] The above is a detailed description of one implementation of the above step S102. Another implementation of the above step S102 will be described below.

[0090] As the map is continuously updated, the accumulated error increases, thereby reducing the accuracy of the 3D point cloud map. To overcome this problem, in some other embodiments of step S102, the vehicle-mounted global satellite positioning device can be reused, taking advantage of the high positioning accuracy of the global satellite positioning device. When the positioning signal of the global satellite positioning device is normal, the global satellite positioning device is used to create a 3D point cloud map to improve the accuracy of the 3D point cloud map. When the positioning signal of the global satellite positioning device is abnormal, the map is updated based on the relative position of the current frame 3D lidar point cloud and the previous map to create a 3D point cloud map.

[0091] A global satellite positioning device refers to a device that uses satellite navigation positioning technology for positioning. In this embodiment, the global satellite positioning device can be a device that performs positioning based on the Global Navigation Satellite System (GNSS), a device that performs positioning based on the Global Positioning System (GPS), a device that performs positioning based on the BeiDou Navigation Satellite System (BDS), or a device based on RTK (Real Time Kinematic) positioning technology. Those skilled in the art can flexibly select different types of global satellite positioning devices according to actual needs, and this embodiment does not specifically limit this.

[0092] The following describes the methods for establishing a three-dimensional point cloud map when the positioning signal of the global satellite positioning device is normal and abnormal.

[0093] (1) Method for establishing a three-dimensional point cloud map when the positioning signal is normal.

[0094] like Figure 2 As shown, the method for establishing a three-dimensional point cloud map in this embodiment mainly includes the following steps S1021 to S1023.

[0095] Step S1021: Determine the pose parameters T for coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle go .

[0096] In practical applications, after installing a global satellite positioning device and a laser radar on a vehicle, it is necessary to convert the pose parameters T between the global satellite positioning device and the laser radar into coordinate systems. go In this embodiment, the position parameters T of the global satellite positioning device can be directly obtained. go The pose parameters T for coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle are obtained from the calibration results go .

[0097] Step S1022: Based on the posture parameter T go According to the global positioning posture obtained by the global satellite positioning device, the radar positioning posture corresponding to each frame of the three-dimensional lidar point cloud of the driving environment collected by the lidar is determined respectively.

[0098] During vehicle driving, the vibration of the vehicle may cause a large error in the radar positioning posture actually output by the lidar. If the radar positioning posture actually output by the lidar is used to build a three-dimensional point cloud map of the driving environment, it will affect the accuracy of the three-dimensional point cloud map.

[0099] Based on the above posture parameter T go The determined radar positioning pose is not the actual radar positioning pose output by the lidar, but rather a radar positioning pose that matches the global positioning pose obtained through coordinate system conversion. Because global satellite positioning devices have relatively high positioning accuracy, the radar positioning pose matched using these pose parameters is also highly accurate. Therefore, using these pose parameters to create a 3D point cloud map of the driving environment greatly improves the accuracy of the 3D point cloud map.

[0100] Step S1023: stitching each frame of the three-dimensional lidar point cloud according to the radar positioning posture to establish a three-dimensional point cloud map of the driving environment.

[0101] Specifically, the relative poses of two adjacent frames of 3D lidar point clouds can be determined based on the radar positioning pose corresponding to each frame of 3D lidar point cloud, and then the two frames of 3D lidar point clouds can be matched based on the relative poses. The two frames of 3D lidar point clouds can be spliced ​​together based on the results of the point cloud matching.

[0102] In some embodiments, in order to improve the accuracy of the three-dimensional point cloud map, each frame of the three-dimensional lidar point cloud can be first subjected to point cloud dedistortion processing, and then the three-dimensional lidar point cloud after point cloud dedistortion processing can be spliced ​​to establish a three-dimensional point cloud map of the driving environment.

[0103] It should be noted that in this embodiment, the conventional point cloud dedistortion method in the field of three-dimensional laser radar point cloud technology can be used to perform point cloud dedistortion processing on each frame of three-dimensional laser radar point cloud respectively, which is not described in detail in this embodiment.

[0104] Based on the method described in steps S1021 to S1023 above, the global satellite positioning device can be used to obtain accurate radar positioning posture, and then the three-dimensional lidar point cloud is spliced ​​according to the radar positioning posture to establish a three-dimensional point cloud map of the driving environment, which can significantly improve the accuracy of the three-dimensional point cloud map.

[0105] The above steps S1021 and S1022 are further explained below.

[0106] First, the above-mentioned step S1021 is explained.

[0107] In order to improve the accuracy of the radar positioning posture obtained by coordinate system conversion based on the posture parameters, in some implementations of the above step S1021, the posture parameters T for coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle can be determined by the following steps 21 to 23.

[0108] Step 21: Based on the pose parameters (initial pose parameters for coordinate system conversion between the global satellite positioning device and the lidar on the vehicle) and the global positioning pose obtained by the global satellite positioning device when the vehicle turns, determine the radar positioning pose corresponding to each frame of the three-dimensional lidar point cloud collected by the lidar when the vehicle turns.

[0109] Radar positioning pose T o It can be expressed as T o =T go *T g , T go Indicates the pose parameters for coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle, T g Indicates the global positioning pose obtained by the global satellite positioning device.

[0110] Compared with straight driving, the position and posture (posture) of the vehicle will change more significantly when turning. Therefore, the data of the vehicle turning can more accurately determine the posture parameter T for the coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle. go .

[0111] Similar to step S1022, based on the above-mentioned posture parameter T go The determined radar positioning pose is not the radar positioning pose actually output by the lidar, but the radar positioning pose obtained by coordinate system conversion that matches the global positioning pose.

[0112] Step 22: Using the point cloud registration algorithm based on the ICP (Iterative Closest Point) algorithm, the pose parameter T is calculated based on the corresponding radar positioning pose of each frame of the 3D lidar point cloud collected by the lidar when the vehicle turns. go Optimize.

[0113] Specifically, a KD-Tree (K-dimensional search tree) of the 3D lidar point cloud is first established; then, based on the KD-Tree, the 3D lidar point clouds with the nearest neighbor matching relationship between two adjacent frames of 3D lidar point clouds are searched; finally, a point cloud registration algorithm based on the ICP (Iterative Closest Point) algorithm is used, and the pose parameters T of the coordinate system transformation between the global satellite positioning device and the lidar on the vehicle are performed based on the 3D lidar point clouds with the nearest neighbor matching relationship. go In some preferred embodiments, the point cloud registration algorithm based on the ICP algorithm can be a point cloud registration algorithm based on the point-to-plane method in the ICP algorithm. For the sake of brevity, the embodiment of the present invention does not specifically describe the point-to-plane method.

[0114] Step 23: The optimized pose parameters T go As the final pose parameter T go , that is, the posture parameter T used in step S1022 go .

[0115] Based on the method described in steps 21 to 23 above, the posture parameter T can be significantly improved. go calibration accuracy.

[0116] The above is a further explanation of step S1021.

[0117] The above step S1022 will be further explained below.

[0118] In order to accurately obtain the radar positioning pose corresponding to each frame of the three-dimensional lidar point cloud, in some implementations of the above step S1022, the radar positioning pose can be obtained through the following steps 31 to 32.

[0119] Step 31: According to the timestamp corresponding to each frame of the three-dimensional lidar point cloud, the global positioning pose obtained by the global satellite positioning device is interpolated to obtain the global positioning pose at each timestamp.

[0120] Because the GPS and LiDAR acquisition times may be out of sync, the LiDAR may sometimes capture a new frame of 3D LiDAR point cloud, but the GPS may not capture the new frame of positioning signals until a slight delay. This means that the 3D LiDAR point cloud actually captured by the LiDAR and the global positioning pose of the positioning signal collected by the GPS are not one-to-one corresponding. However, by using the above-mentioned method of timestamp interpolation of the global positioning pose, we can accurately determine the global positioning pose corresponding to each frame of the 3D LiDAR point cloud, effectively obtaining a one-to-one global positioning pose corresponding to each frame of the 3D LiDAR point cloud actually captured by the LiDAR.

[0121] Step 32: Pose parameters T based on the coordinate system conversion between the global satellite positioning device and the lidar go , and according to the global positioning pose at each timestamp, the corresponding radar positioning pose of each frame of 3D lidar point cloud is determined respectively.

[0122] Based on the method described in steps 31 to 32 above, the radar positioning pose corresponding to each frame of three-dimensional lidar point cloud actually collected by the lidar can be accurately obtained.

[0123] The above is a further explanation of step S1022.

[0124] (2) A method for establishing a three-dimensional point cloud map when the positioning signal is abnormal.

[0125] Similar to the implementation of the aforementioned step S102, when the positioning signal of the global satellite positioning device is abnormal, a three-dimensional point cloud map of the driving environment can be established in the following manner.

[0126] For each frame of the 3D LiDAR point cloud, the relative pose of the current frame's point cloud relative to the 3D point cloud map built from the previous multiple frames of 3D LiDAR point cloud is determined. Based on this relative pose and the current frame's point cloud, the 3D point cloud map is updated to create a 3D point cloud map of the driving environment. For simplicity, the "3D point cloud map built from the previous multiple frames of 3D LiDAR point cloud" will be referred to as the "prior map."

[0127] According to step 15 in the aforementioned embodiment, when determining the relative pose of the current frame three-dimensional lidar point cloud with respect to the previous map, the geometric structural features of the point cloud (the distance between the line segment formed by the previously matched edge point cloud and the second edge feature point and the distance between the previously matched plane point cloud and the plane formed by the second plane feature point) are mainly aligned to determine the relative pose. However, when the vehicle is traveling in a scene with a strong repetitiveness of geometric structural features (such as a highway, tunnel or cross-sea bridge, etc.), the above-mentioned method of aligning the geometric structural features of the point cloud to determine the relative pose may cause mismatching. In this regard, in order to overcome the above-mentioned problem and improve the robustness of the method for determining the relative pose, in this embodiment, the relative pose of the current frame three-dimensional lidar point cloud with respect to the previous map can be determined by the following steps 41 to 42.

[0128] Step 41: Based on the angular velocity and linear velocity of the vehicle, predict the relative pose of the current frame 3D lidar point cloud relative to the 3D point cloud map established based on the previous multiple frames of 3D lidar point clouds, and use the predicted relative pose as the prior relative pose.

[0129] Angular velocity refers to the arc traveled per unit time (e.g., 1 second), while linear velocity refers to the distance traveled per unit time (e.g., 1 second). When the GPS positioning signal is abnormal, the angular velocity and linear velocity can be received from the vehicle's wheel odometer via the CAN (Controller Area Network) bus. When the GPS positioning signal is normal, the angular velocity and linear velocity can be obtained from the GPS.

[0130] The angular velocity and linear velocity of the vehicle usually do not change suddenly within one or several scanning cycles (short time) of the lidar. Therefore, the posture change of the current frame 3D lidar point cloud can be predicted based on the angular velocity of the vehicle in the scanning cycle corresponding to the previous frame or the previous few frames of the 3D lidar point cloud. At the same time, the position change of the current frame 3D lidar point cloud can also be predicted based on the linear velocity of the vehicle in the scanning cycle corresponding to the previous frame or the previous few frames of the 3D lidar point cloud. Based on the above posture change and attitude change, the prediction result of the relative posture of the current frame 3D lidar point cloud relative to the previous map can be obtained.

[0131] Step 42: Based on the prior relative pose, determine the relative pose of the current frame 3D lidar point cloud relative to the 3D point cloud map established based on the previous multiple frames of 3D lidar point clouds.

[0132] Specifically, the relative pose of the current frame's 3D LiDAR point cloud relative to the prior map is first determined using the method described in steps 11 to 16 of the aforementioned embodiment. For simplicity, this relative pose is referred to as the initial relative pose. The initial relative pose is then compared with the prior relative pose. If the difference between the two is small, the initial relative pose is used as the final relative pose. If the difference between the two is large, the prior relative pose is used as the final relative pose.

[0133] Based on the method described in steps 41 to 42 above, even when the vehicle is driving in a scene with highly repetitive geometric structure features, the relative position of the current frame three-dimensional lidar point cloud relative to the previous map can be accurately obtained.

[0134] In addition, in other embodiments, in order to further improve the robustness of the method for determining relative pose, before determining the relative pose of the current frame three-dimensional lidar point cloud relative to the previous map, dynamic object detection can be performed on the current frame three-dimensional lidar point cloud, and the three-dimensional lidar point cloud belonging to dynamic objects in the current frame three-dimensional lidar point cloud can be removed according to the detection results. Finally, the relative pose is determined based on the current frame three-dimensional lidar point cloud after removing the three-dimensional lidar point cloud belonging to dynamic objects.

[0135] In this embodiment, 3D lidar point cloud samples can be used to train a dynamic object detection model based on a deep convolutional neural network, and then the trained dynamic object detection model is used to perform dynamic object detection on each frame of 3D lidar point cloud. The dynamic object detection model can output a detection frame for each dynamic object in each frame of 3D lidar point cloud. The 3D lidar point cloud within the detection frame is the 3D lidar point cloud belonging to the dynamic object, and these 3D lidar point clouds can be removed.

[0136] Furthermore, in this embodiment, in order to avoid drift of the radar positioning posture and / or three-dimensional point cloud map corresponding to the current frame three-dimensional lidar point cloud actually output by the lidar, after updating the three-dimensional point cloud map based on the relative posture of the current frame three-dimensional lidar point cloud relative to the previous map and the current frame three-dimensional lidar point cloud to establish a three-dimensional point cloud map of the driving environment, the radar positioning posture corresponding to the current frame three-dimensional lidar point cloud can also be optimized through the following step 51.

[0137] Step 51: According to the constraints of the preset factor graph (Pose Graph), a pose optimization method based on the factor graph (Pose Graph) is used to optimize the radar positioning pose corresponding to the current frame 3D lidar point cloud. It should be noted that the pose optimization method based on the factor graph (Pose Graph) is a conventional pose optimization method in the field of positioning and mapping technology. This embodiment does not explain the specific principles of this method, but only explains the constraints used by this method.

[0138] The constraints of the preset factor graph (Pose Graph) may include radar positioning pose constraints and global positioning pose constraints. The radar positioning pose constraints and global positioning pose constraints are described below.

[0139] (1) Radar positioning posture constraints.

[0140] In this embodiment, the radar positioning posture constraint condition is shown in the following formula (1).

[0141]

[0142] The meanings of the parameters in formula (1) are as follows:

[0143] r odom represents the constraint error of the radar positioning pose constraint condition, represents the absolute pose of the i-1th key frame in the Pose Graph, represents the absolute pose of the i-th key frame in the Pose Graph, Express Perform the inverse operation, represents the relative pose of the i-th keyframe relative to the i-1-th keyframe, w represents the world coordinate system, and b represents the vehicle coordinate system. The absolute pose refers to the pose relative to the world coordinate system.

[0144] A keyframe is a frame of 3D LiDAR point cloud where the change in radar pose compared to the previous frame is greater than a preset threshold. This means that when building the Pose Graph, the Pose Graph is constructed based on those 3D LiDAR point clouds with large changes in radar pose, while those with small changes are removed. Because 3D LiDAR point clouds with small changes in radar pose have a relatively small impact on vehicle positioning and map updates, and these 3D LiDAR point clouds are also relatively numerous, removing these 3D LiDAR point clouds with small changes in radar pose significantly improves the efficiency of pose optimization based on the Pose Graph.

[0145] (2) Global positioning posture constraints.

[0146] In this embodiment, the global positioning posture constraint condition is shown in the following formula (2).

[0147]

[0148] The meanings of the parameters in formula (2) are as follows:

[0149] r rtk represents the constraint error of the global positioning pose constraint, represents the global positioning pose at the timestamp corresponding to the j-th key frame in the Pose Graph, T rb Indicates the pose parameters for the coordinate system conversion between the vehicle coordinate system and the global satellite positioning device. represents the absolute pose of the jth keyframe in the Pose Graph, Express Perform an inverse operation, r represents the coordinate system of the global satellite positioning device, j∈Ω, and Ω represents the key frame in the three-dimensional lidar point cloud collected by the lidar when the positioning signal of the global satellite positioning device is normal.

[0150] The constraint objective of the above-mentioned preset factor graph (Pose Graph) constraint condition is to make the constraint error r odom and constraint error r rtk The sum is minimized.

[0151] The above is Figure 1 Detailed description of step S102 of the method embodiment is shown.

[0152] So far, the first embodiment of the method for obtaining lane marking data has been described with reference to the accompanying drawings. Now, the second embodiment of the method for obtaining lane marking data will be described.

[0153] 2. Second Embodiment of the Method for Acquiring Lane Marking Data

[0154] In the method for obtaining lane marking data according to the second embodiment of the present invention, the method for obtaining lane marking data may include: Figure 1 Steps S101 to S104 of the method embodiment shown in FIG. Figure 1 The main difference between the embodiment of the method shown is that after executing step S102 and before executing step S103, it also includes a step of removing the 3D LiDAR point cloud belonging to dynamic objects. Figure 3 , Figure 3 The main process of the method for removing the three-dimensional laser radar point cloud belonging to dynamic objects in an embodiment of the present invention is exemplified. Figure 3 As shown, in an embodiment of the present invention, the three-dimensional lidar point cloud belonging to a dynamic object can be removed through the following steps S201 to S202.

[0155] Step S201: performing ground fitting on the 3D lidar point cloud in the 3D point cloud map to determine the ground plane of the 3D point cloud map.

[0156] Step S202: removing the three-dimensional lidar point cloud belonging to dynamic objects in the three-dimensional point cloud map according to the ground plane.

[0157] Specifically, a 3D LiDAR point cloud located above the ground plane can be regarded as a 3D LiDAR point cloud belonging to a dynamic object. In some embodiments, after determining the ground plane, the ground height at the position corresponding to each 3D LiDAR point cloud can be used to determine whether the 3D LiDAR point cloud is a ground point cloud or a non-ground point cloud based on the point cloud height and the ground height of each 3D LiDAR point cloud. Finally, all non-ground point clouds are removed. If the point cloud height is less than or equal to the ground height, the 3D LiDAR point cloud is a ground point cloud; otherwise, the 3D LiDAR point cloud is a non-ground point cloud.

[0158] Based on the method described in steps S201 to S202 above, the three-dimensional lidar point cloud belonging to dynamic objects in the three-dimensional point cloud map can be removed. In this way, when lane elements are labeled according to the three-dimensional point cloud map, the interference of the three-dimensional lidar point cloud belonging to dynamic objects on the lane elements can be eliminated, thereby improving the labeling efficiency of the lane elements.

[0159] The above step S201 is further explained below.

[0160] See attached Figure 4 In some implementations of the above step S201, the ground plane of the three-dimensional point cloud map can be determined through the following steps S2011 to S2014.

[0161] Step S2011: using a polar coordinate grid representation method based on a concentric zone model, with the center point of the three-dimensional point cloud map as the pole, to establish a polar coordinate grid map of the three-dimensional point cloud map.

[0162] by Figure 5 Taking the polar coordinate grid diagram shown in FIG. 1 as an example, the polar coordinate grid representation method based on the concentric zone model is briefly explained.

[0163] Specifically, with the center point of the three-dimensional point cloud map as the pole, multiple concentric circle areas are formed along the order of the polar diameter values ​​from small to large. The polar diameter values ​​corresponding to each concentric circle area can be the same or different. Figure 5 As shown, four concentric circle areas Z1, Z2, Z3 and Z4 are formed in the order of polar diameter values ​​from small to large, and the polar diameter values ​​corresponding to each of them are different. After determining multiple concentric circle areas, each concentric circle area is gridded again. In the same concentric circle area, the polar angle angle corresponding to each grid (the range of polar angle angles covered by the grid) is the same, and the polar angle angles corresponding to the grids in different concentric circle areas may be different. Figure 5 As shown, the polar angles corresponding to the grids in the concentric circle areas Z2 and Z3 are the same and the smallest, followed by the concentric circle area Z1 with the smallest polar angle, and the concentric circle area Z4 with the largest polar angle.

[0164] Step S2012: For each grid in the polar coordinate grid diagram, plane fitting is performed on the three-dimensional lidar point cloud within the grid to obtain multiple planes within the grid.

[0165] In this embodiment, a conventional plane fitting method in the field of plane fitting technology can be used to perform plane fitting on the three-dimensional lidar point cloud in each grid. For example, a RANSAC (RANdom Sample Consensus) algorithm can be used for plane fitting.

[0166] Step S2013: Determine whether the grid is a ground grid based on the plane normal vector of each plane and the height of each three-dimensional lidar point cloud in the grid, and with the constraint conditions that the vertical angle deviation of each plane normal vector is less than a preset angle deviation threshold and the height difference between adjacent three-dimensional lidar point clouds in the grid is less than a preset height difference threshold.

[0167] Vertical angle deviation refers to the angular deviation between the plane normal vector and the vertical direction of the 3D LiDAR point cloud. The vector [0 0 1] can be used to represent vertically upward, or the vector [0 0 -1] can be used to represent vertically downward. After determining the plane normal vector, the vector angle between the plane normal vector and the vertically upward vector [0 0 1] (or vertically downward vector [0 0 -1]) can be calculated using the vector angle calculation method. This vector angle is used as the vertical angle deviation.

[0168] If the vertical angle deviation is less than the preset angle deviation threshold, it indicates that the plane normal vector is perpendicular to the ground, and the plane corresponding to the plane normal vector is likely the ground plane. If the height difference between adjacent 3D LiDAR point clouds within a grid is less than the preset height difference threshold, it indicates that the 3D LiDAR point cloud within the grid is relatively smooth, and the grid may also be a ground grid. If both the vertical angle deviation is less than the preset angle deviation threshold and the height difference between adjacent 3D LiDAR point clouds within the grid is less than the preset height difference threshold are met, the grid can be determined to be a ground grid.

[0169] Step S2014: performing ground fitting on the three-dimensional lidar point cloud within the ground grid to determine the ground plane of the three-dimensional point cloud map.

[0170] In this embodiment, a conventional plane fitting method in the field of plane fitting technology can be used to perform ground fitting on the three-dimensional lidar point cloud within the ground grid to determine the ground plane of the three-dimensional point cloud map, which is not described in detail in this embodiment.

[0171] Based on the method described in steps S2011 to S2014 above, the ground plane of the three-dimensional point cloud map can be accurately obtained, thereby improving the accuracy of the ground point cloud and non-ground point cloud, and ultimately improving the accuracy of removing the three-dimensional lidar point cloud belonging to dynamic objects.

[0172] So far, the second embodiment of the method for obtaining lane marking data has been described with reference to the accompanying drawings. Now, the third embodiment of the method for obtaining lane marking data will be described.

[0173] 3. Third Embodiment of the Method for Acquiring Lane Marking Data

[0174] In the method for obtaining lane marking data according to the third embodiment of the present invention, the method for obtaining lane marking data includes: Figure 1 Steps S101 to S104 of the method embodiment shown in FIG. Figure 1The main difference between the illustrated method embodiment is that, after executing step S102 and before executing step S103, the method further includes a step of coloring the 3D LiDAR point cloud. Specifically, the step of coloring the 3D LiDAR point cloud may include:

[0175] The three-dimensional point cloud map of the driving environment is colored based on the color information of the two-dimensional image of the driving environment captured by the image acquisition device on the vehicle, so that the driving environment can be visualized when generating a bird's eye view of the driving environment.

[0176] Specifically, the color information of the two-dimensional image can be added to the point cloud information of the three-dimensional lidar point cloud in the three-dimensional point cloud map to achieve coloring processing of the three-dimensional point cloud map.

[0177] In some preferred implementations, the three-dimensional point cloud map of the driving environment may be colored through the following steps S301 to S302.

[0178] Step S301: Determine clear 3D lidar point clouds and unclear 3D lidar point clouds according to the laser reflection intensity of each 3D lidar point cloud in the 3D point cloud map.

[0179] Specifically, the laser reflection intensity can be compared with a preset intensity threshold. If the laser reflection intensity is less than the preset intensity threshold, the 3D LiDAR point cloud is unclear. If the laser reflection intensity is greater than or equal to the preset intensity threshold, the 3D LiDAR point cloud is clear. Those skilled in the art can flexibly set the specific value of the preset intensity threshold based on actual needs, and this embodiment does not specifically limit this.

[0180] In some preferred embodiments, clear three-dimensional lidar point clouds and unclear three-dimensional lidar point clouds in each frame of three-dimensional lidar point cloud of the three-dimensional point cloud map can be determined separately by parallel processing to improve processing efficiency.

[0181] Step S302: Coloring the unclear three-dimensional lidar point cloud according to the color information of the two-dimensional image of the driving environment, so that the unclear three-dimensional lidar point cloud can be visualized when generating a bird's eye view of the driving environment.

[0182] Specifically, the color information of the two-dimensional image can be added to the point cloud information of the unclear three-dimensional lidar point cloud to achieve coloring processing of the three-dimensional point cloud map.

[0183] Similar to step S301, in some preferred embodiments, the unclear three-dimensional lidar point cloud in each frame of the three-dimensional lidar point cloud can be colored separately according to the color information of the two-dimensional image of the driving environment captured by the image acquisition device on the vehicle through parallel processing to improve processing efficiency.

[0184] Based on the method described in steps S301 to S302 above, the unclear three-dimensional lidar point cloud can be colored to improve the visualization of the unclear three-dimensional lidar point cloud.

[0185] The above step S302 is further explained below.

[0186] In some implementations of the above step S302, the unclear three-dimensional lidar point cloud can be colored through the following steps S3021 to S3022.

[0187] Step S3021: Determine the nearest neighbor two-dimensional image corresponding to each frame of three-dimensional lidar point cloud according to the timestamp corresponding to each frame of three-dimensional lidar point cloud and the timestamp corresponding to each frame of two-dimensional image collected by the image acquisition device.

[0188] Step S3022: performing coloring processing on unclear three-dimensional lidar point clouds in each frame of three-dimensional lidar point cloud according to the nearest neighbor two-dimensional image.

[0189] Specifically, first, according to the timestamp corresponding to the nearest neighbor two-dimensional image, the radar positioning pose obtained by the lidar is interpolated and calculated to obtain the radar positioning pose at the timestamp; then, based on the pose parameters of the coordinate system conversion between the lidar and the image acquisition device, and according to the radar positioning pose at the above timestamp, the image positioning pose corresponding to the nearest neighbor two-dimensional image is determined; then, according to the image positioning pose and the position information of the unclear three-dimensional lidar point cloud in the radar coordinate system, the projected image point after the unclear three-dimensional lidar point cloud is projected onto the nearest neighbor two-dimensional image is determined; finally, the unclear three-dimensional lidar point cloud is colored according to the color information of this projected image point.

[0190] Furthermore, when there are multiple image acquisition devices on the vehicle and the nearest neighbor two-dimensional image corresponding to the current frame three-dimensional lidar point cloud includes two-dimensional images acquired by multiple image acquisition devices, the two-dimensional image acquired by the last image acquisition device can be selected as the final nearest neighbor two-dimensional image according to the preset arrangement order of the image acquisition devices, and then the unclear three-dimensional lidar point cloud in the current frame three-dimensional lidar point cloud can be colored according to the final nearest neighbor two-dimensional image.

[0191] Assume that six image acquisition devices A, B, C, D, E and F are provided on the vehicle, and the arrangement order of these six image acquisition devices is ABCDEF.

[0192] For example, the nearest neighbor two-dimensional image corresponding to the three-dimensional lidar point cloud of the current frame includes the two-dimensional images a, b, c, d, e and f captured by the six image acquisition devices A, B, C, D, E and F. Since the last image acquisition device is F, the two-dimensional image f is used as the final nearest neighbor two-dimensional image.

[0193] For another example, the nearest neighbor two-dimensional image corresponding to the three-dimensional lidar point cloud of the current frame includes the two-dimensional images a, c and e captured by the three image acquisition devices A, C and E. Since the last image acquisition device is E, the two-dimensional image e is used as the final nearest neighbor two-dimensional image.

[0194] Based on the method described in steps S3021 to S3022 above, even when multiple image acquisition devices are installed on a vehicle, unclear three-dimensional lidar point clouds can be accurately and reliably colored.

[0195] In other implementations of the above step S302, the unclear three-dimensional lidar point cloud can be colored through the following steps S3023 to S3024.

[0196] Step S3023: extracting a preset driving environment region of interest (Region of Interest) from the two-dimensional image.

[0197] The preset driving environment region of interest (ROI) includes at least the lane area in the driving environment and excludes areas that may interfere with point cloud shading. The areas that may interfere with point cloud shading include at least the vehicle image area and areas of interference caused by image exposure. The areas of interference caused by image exposure may be areas of interference caused by the rolling shutter effect.

[0198] Step S3024: coloring the unclear 3D lidar point cloud according to the color information of the preset driving environment region of interest.

[0199] The implementation of this step is similar to the method described in the aforementioned step S3022 and will not be repeated here.

[0200] Furthermore, in the method for obtaining lane marking data according to the third embodiment of the present invention, the method for obtaining lane marking data includes: Figure 1 Step S103 and step S104 of the illustrated method embodiment may be replaced by the following steps S303 to S305 , respectively.

[0201] Step S303: Generate a first bird's eye view based on the clear 3D lidar point cloud in the colored 3D point cloud map, generate a second bird's eye view based on the unclear 3D lidar point cloud in the colored 3D point cloud map, and generate a height map based on the colored 3D point cloud map.

[0202] In the first bird's eye view, each image point corresponds one-to-one to a clear three-dimensional lidar point cloud; in the second bird's eye view, each image point corresponds one-to-one to an unclear three-dimensional lidar point cloud.

[0203] Step S304: In response to receiving the marking start instruction, loading and displaying a first bird's eye view and a height map.

[0204] Step S305: During the process of loading the first bird's eye view and the height map, if a label switching instruction is received, the second bird's eye view and the height map are loaded and displayed.

[0205] Based on the method described in steps S303 to S305 above, the first bird's eye view can be loaded first when starting the lane element labeling work. When certain lane elements cannot be clearly seen from the first bird's eye view, the second bird's eye view can be switched to load to continue completing the lane element labeling work.

[0206] Furthermore, in the method for obtaining lane marking data according to the third embodiment of the present invention, the method for obtaining lane marking data may include: Figure 1 Steps S101 to S104 of the method embodiment shown may also include Figure 3 Steps S201 to S202 of the method embodiment shown (the method for obtaining lane marking data of the second embodiment) At this time, the step of coloring the 3D lidar point cloud can be performed after executing step S202 and before executing step S103.

[0207] So far, the third embodiment of the method for obtaining lane marking data has been described with reference to the accompanying drawings. Now, the fourth embodiment of the method for obtaining lane marking data will be described.

[0208] 4. Fourth Embodiment of the Method for Acquiring Lane Marking Data

[0209] In the method for obtaining lane marking data according to the fourth embodiment of the present invention, the method for obtaining lane marking data may include: Figure 1 Steps S101 to S104 of the method embodiment shown in FIG. Figure 1 The main difference between the embodiment of the method shown is that after executing step S104, it also includes a step of calibrating the posture parameters of the laser radar and the image acquisition device for coordinate system conversion. Figure 6 , Figure 6 The main process of the method for calibrating the pose parameters of the laser radar and the image acquisition device for coordinate system conversion in an embodiment of the present invention is exemplified. Figure 6 As shown, in an embodiment of the present invention, the posture parameters of the laser radar and the image acquisition device for coordinate system conversion can be calibrated through the following steps S401 to S402.

[0210] Step S401: Acquire two-dimensional lane element detection data obtained by performing lane element detection on a two-dimensional image of a driving environment captured by an image capture device on a vehicle.

[0211] In this embodiment, two-dimensional image samples can be used to train a two-dimensional lane element detection model based on a deep convolutional neural network, and then the trained two-dimensional lane element detection model is used to perform lane element detection on the two-dimensional image.

[0212] Step S402: Calibrate the pose parameters of the laser radar and the image acquisition device for coordinate system conversion based on the four-dimensional lane element annotation data and the two-dimensional lane element detection data.

[0213] See attached Figure 7 In this embodiment, the pose parameters of the laser radar and the image acquisition device for coordinate system conversion can be calibrated through the following steps S4021 to S4024.

[0214] Step S4021: Project the four-dimensional lane element annotation data to the coordinate system of the image acquisition device according to the posture parameters to obtain the projection point f(A, O) of the four-dimensional lane element annotation data.

[0215] Where A represents the coordinates of the four-dimensional lane element annotation data in the laser radar coordinate system, and O represents the pose parameters.

[0216] Step S4022: Determine the two image points on the two-dimensional image that are closest to the projection point f(A, O).

[0217] Step S4023: Based on the line segment cd formed by the projection point f(A, O) and the two image points, establish a distance error equation from the projection point f(A, O) to the line segment cd.

[0218] The distance error equation from the projection point f(A,O) to the line segment cd is shown in equation (3).

[0219] loss=d(f(A,O),cd) (3)

[0220] The meanings of the parameters in the above formula (3) are as follows:

[0221] d represents the distance calculation function from the projection point f(A,O) to the line segment cd, and loss represents the distance calculated by the distance calculation function d.

[0222] Step S4024: With the goal of making the distance loss less than the preset distance threshold, iteratively optimize the posture parameter O in the distance error equation, and obtain the posture parameter O when the distance loss is less than the preset distance threshold, and use the posture parameter O as the calibrated posture parameter.

[0223] In the embodiment of the present invention, a least squares algorithm can be used to iteratively optimize the pose parameter O in the distance error equation. For example, the Levenberg-Marquardt algorithm in the least squares algorithm can be used to iteratively optimize the pose parameter O in the distance error equation. The embodiment of the present invention does not specifically limit the above iterative optimization method.

[0224] Based on the method described in steps S4021 to S4024 above, the accuracy of calibrating the posture parameters of the laser radar and the image acquisition device for coordinate system conversion can be improved.

[0225] Furthermore, in the lane marking data acquisition method according to the fourth embodiment of the present invention, in addition to including the aforementioned steps S401 to S402, the lane marking data acquisition method can also determine the lane element annotation data on the two-dimensional image of the driving environment based on the calibrated posture parameters and according to the four-dimensional lane element annotation data.

[0226] Specifically, the calibrated pose parameters can be used to transform the coordinate system of the 4D lane element annotation data, determining the image points corresponding to the 4D lane element annotation data on the 2D image. Based on these image points, the lane element annotation data on the 2D image can be determined. This method can greatly improve the efficiency and accuracy of obtaining lane element annotation data on the 2D image.

[0227] The above is a description of the fourth embodiment of the method for acquiring lane marking data.

[0228] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0229] Those skilled in the art will appreciate that all or part of the processes in the method for implementing the above-mentioned embodiment of the present invention may also be accomplished by instructing the relevant hardware through a computer program. The computer program may be stored in a computer-readable storage medium. When the computer program is executed by a processor, it may implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal, and software distribution medium capable of carrying the computer program code. It should be noted that the content contained in the computer-readable storage medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0230] Furthermore, the present invention also provides a computer device.

[0231] In an embodiment of a computer device according to the present invention, the computer device includes a processor and a storage device. The storage device can be configured to store a program for executing the lane marking data acquisition method of the above-described method embodiment, and the processor can be configured to execute the program in the storage device, including but not limited to a program for executing the lane marking data acquisition method of the above-described method embodiment. For ease of illustration, only the portions relevant to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method section of the embodiment of the present invention. The computer device can be a control device formed by various electronic devices.

[0232] Furthermore, the present invention also provides a computer-readable storage medium.

[0233] In one embodiment of a computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program for executing the lane marking data acquisition method of the above-described method embodiment. This program can be loaded and executed by a processor to implement the lane marking data acquisition method described above. For ease of illustration, only the portions relevant to the embodiments of the present invention are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, in embodiments of the present invention, the computer-readable storage medium is non-transitory.

[0234] Thus far, the technical solution of the present invention has been described in conjunction with an embodiment shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A method for obtaining lane marking data, characterized in that: The method comprises: Acquire a three-dimensional laser radar point cloud of the driving environment collected by the laser radar on the vehicle during the vehicle's driving process; Creating a three-dimensional point cloud map of the driving environment based on the three-dimensional laser radar point cloud; generating a bird's-eye view and a height map of the driving environment respectively according to the three-dimensional point cloud map of the driving environment; Annotating lane elements of the driving environment during vehicle driving according to the bird's-eye view and the height map to form four-dimensional lane element annotation data including one-dimensional time information and three-dimensional spatial position information; After forming the four-dimensional lane element annotation data, the method further includes: obtaining two-dimensional lane element detection data obtained by performing lane element detection on a two-dimensional image of the driving environment captured by an image acquisition device on the vehicle; calibrating the pose parameters of the laser radar and the image acquisition device for coordinate system conversion based on the four-dimensional lane element annotation data and the two-dimensional lane element detection data; wherein the pose parameter calibration includes: According to the pose parameters, the four-dimensional lane element annotation data is projected onto the coordinate system of the image acquisition device to obtain the projection point f(A,O) , A Represents the coordinates of the four-dimensional lane element annotation data in the laser radar coordinate system. O represents the posture parameters; Determine the distance projection point on the two-dimensional image f(A,O) The two nearest image points; according to the projection point f(A,O) The line segment formed by the two image points cd , establish the distance error equation loss=d(f(A,O),cd) , d Represents the projection point f(A,O) To line segment cd The distance calculation function, loss Represents the distance calculation function d The calculated distance; so that the distance loss The distance is less than the preset threshold value, and the pose parameters in the distance error equation are O Perform iterative optimization and obtain the distance loss Pose parameters when the distance is less than the preset threshold O , the pose parameters O As the calibrated pose parameters.

2. The method for obtaining lane marking data according to claim 1, characterized in that: Before the step of "generating a bird's-eye view and a height map of the driving environment based on the three-dimensional point cloud map of the driving environment", the method further includes: performing ground fitting on the three-dimensional lidar point cloud in the three-dimensional point cloud map to determine a ground plane of the three-dimensional point cloud map; The three-dimensional lidar point cloud belonging to a dynamic object in the three-dimensional point cloud map is removed according to the ground plane.

3. The method for obtaining lane marking data according to claim 2, characterized in that: The step of “performing ground fitting on the three-dimensional lidar point cloud in the three-dimensional point cloud map to determine the ground plane of the three-dimensional point cloud map” specifically includes: Using a polar coordinate grid representation method based on a concentric circle model, with the center point of the three-dimensional point cloud map as the pole, a polar coordinate grid diagram of the three-dimensional point cloud map is established; For each grid in the polar coordinate grid diagram, performing plane fitting on the three-dimensional lidar point cloud within the grid to obtain multiple planes within the grid; Determining whether the grid is a ground grid based on the plane normal vector of each plane and the height of each three-dimensional lidar point cloud in the grid, and subjecting to the constraints that the vertical angle deviation of each plane normal vector is less than a preset angle deviation threshold and the height difference between adjacent three-dimensional lidar point clouds in the grid is less than a preset height difference threshold; Ground fitting is performed on the three-dimensional lidar point cloud within the ground grid to determine the ground plane of the three-dimensional point cloud map.

4. The method for obtaining lane marking data according to claim 1, characterized in that: Before the step of "generating a bird's-eye view and a height map of the driving environment based on the three-dimensional point cloud map of the driving environment", the method further includes: The three-dimensional point cloud map of the driving environment is colored according to the color information of the two-dimensional image of the driving environment captured by the image acquisition device on the vehicle, so that the driving environment can be visualized when generating a bird's-eye view of the driving environment.

5. The method for obtaining lane marking data according to claim 4, characterized in that: The step of "coloring the three-dimensional point cloud map of the driving environment according to the color information of the two-dimensional image of the driving environment captured by the image capture device on the vehicle, so as to be able to visualize the driving environment when generating a bird's-eye view of the driving environment" specifically includes: Determining a clear 3D lidar point cloud and an unclear 3D lidar point cloud according to the laser reflection intensity of each 3D lidar point cloud in the 3D point cloud map; The unclear three-dimensional lidar point cloud is colored according to the color information of the two-dimensional image of the driving environment, so that the unclear three-dimensional lidar point cloud can be visualized when generating a bird's-eye view of the driving environment.

6. The method for obtaining lane marking data according to claim 1, characterized in that: The step of “establishing a three-dimensional point cloud map of the driving environment based on the three-dimensional laser radar point cloud” specifically includes: For each frame of three-dimensional lidar point cloud, determine the relative pose of the current frame three-dimensional lidar point cloud relative to the three-dimensional point cloud map established based on the previous multiple frames of three-dimensional lidar point clouds, and update the three-dimensional point cloud map based on the relative pose and the current frame three-dimensional lidar point cloud to establish a three-dimensional point cloud map of the driving environment.

7. The method for acquiring lane marking data according to claim 1, characterized in that: After the step of "calibrating the pose parameters for coordinate system conversion of the laser radar and the image acquisition device based on the four-dimensional lane element annotation data and the two-dimensional lane element detection data", the method further includes: Based on the calibrated pose parameters and according to the four-dimensional lane element annotation data, lane element annotation data on the two-dimensional image of the driving environment is determined.

8. A computer device comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, wherein: The program code is suitable for being loaded and run by the processor to execute the method for acquiring lane marking data according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the method for acquiring lane marking data according to any one of claims 1 to 7.

Images