Point cloud map processing method, lane marking data acquisition method, device and medium

Abstract

The present invention relates to the field of autonomous driving technology, and specifically provides a point cloud map processing method, a lane marking data acquisition method, a device and a medium, aiming to solve the problem of improving the accuracy of lane marking data. To this end, the point cloud map processing method provided by the present invention includes establishing a three-dimensional point cloud map of the driving environment based on the three-dimensional laser radar point cloud of the driving environment during the driving of the vehicle, determining unclear three-dimensional laser radar point clouds based on the laser reflection intensity of each three-dimensional laser radar point cloud in the three-dimensional point cloud map; and coloring the unclear three-dimensional laser radar point cloud based on the color information of the two-dimensional image of the driving environment collected by the image acquisition device on the vehicle for visual display. Through the above method, the unclear three-dimensional laser radar point cloud in the driving environment can be visualized, and the visualization degree of the three-dimensional point cloud map can be improved, so that the annotation efficiency and accuracy of the lane elements can be improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a point cloud map processing method, a lane marking data acquisition method, a device and a medium. Background Art

[0002] Autonomous driving technology mainly includes three key technologies: perception, planning and control. Among them, perception technology is mainly used to determine the real-time local map and obstacle information of the vehicle's driving environment. Planning technology is mainly used to plan the vehicle's driving trajectory based on the above real-time local map and obstacle information. Control technology is mainly used to control the vehicle to drive according to the planned driving trajectory.

[0003] In order to improve the accuracy of real-time local maps and obstacle information to ensure safe driving of vehicles, the current method is to first use lane annotation data of lane scenes to train a multi-sensor perception model based on the BEV (Bird Eye View) perspective, and then determine the real-time local map and obstacle information of the vehicle's driving environment through the trained perception model. Among them, the accuracy of lane annotation data will greatly affect the accuracy of the perception model, and then affect the accuracy of real-time local maps and obstacle information. Therefore, in order to improve the accuracy of lane annotation data, improve the accuracy of real-time local maps and obstacle information, and then ensure safe driving of vehicles, it is necessary to accurately obtain lane annotation data of lane scenes.

[0004] Accordingly, the art needs a new technical solution to solve the above problems. Summary of the invention

[0005] In order to overcome the above-mentioned defects, the present invention is proposed to provide a point cloud map processing method, lane marking data acquisition method, device and medium that solve or at least partially solve the technical problem of how to improve the accuracy of lane marking data, so as to improve the accuracy of real-time local maps and obstacle information, and thereby ensure the safe driving of vehicles.

[0006] In a first aspect, a method for processing a three-dimensional point cloud map is provided, the method comprising:

[0007] Acquire a three-dimensional laser radar point cloud of the driving environment collected by the laser radar on the vehicle during the driving process of the vehicle; establish a three-dimensional point cloud map of the driving environment based on the three-dimensional laser radar point cloud; determine clear three-dimensional laser radar point clouds and unclear three-dimensional laser radar point clouds based on the laser reflection intensity of each three-dimensional laser radar point cloud in the three-dimensional point cloud map; and colorize the unclear three-dimensional laser radar point cloud based on the color information of the two-dimensional image of the driving environment collected by the image acquisition device on the vehicle for visual display.

[0008] In a technical solution of the above-mentioned three-dimensional point cloud map processing method, the step of "coloring the unclear three-dimensional lidar point cloud for visual display according to the color information of the two-dimensional image of the driving environment collected by the image acquisition device on the vehicle" specifically includes: determining the nearest neighbor two-dimensional image corresponding to each frame of the three-dimensional lidar point cloud according to the timestamp corresponding to each frame of the three-dimensional lidar point cloud and the timestamp corresponding to each frame of the two-dimensional image collected by the image acquisition device; and coloring the unclear three-dimensional lidar point cloud in each frame of the three-dimensional lidar point cloud according to the nearest neighbor two-dimensional image.

[0009] In a technical solution of the above-mentioned three-dimensional point cloud map processing method, there are multiple image acquisition devices on the vehicle, and the step of "coloring the unclear three-dimensional lidar point cloud in each frame of three-dimensional lidar point cloud according to the nearest neighbor two-dimensional image" specifically includes: for each frame of three-dimensional lidar point cloud, if the nearest neighbor two-dimensional image corresponding to the current frame three-dimensional lidar point cloud includes two-dimensional images acquired by multiple image acquisition devices, then according to a preset arrangement order of the image acquisition devices, the last two-dimensional image acquired by the image acquisition device is selected as the final nearest neighbor two-dimensional image; and the unclear three-dimensional lidar point cloud in the current frame three-dimensional lidar point cloud is colored according to the final nearest neighbor two-dimensional image.

[0010] In a technical solution of the above-mentioned three-dimensional point cloud map processing method, the step of "coloring the unclear three-dimensional lidar point cloud for visual display according to the color information of the two-dimensional image of the driving environment collected by the image acquisition device on the vehicle" also includes: extracting a preset driving environment region of interest (Region of Interest) in the two-dimensional image; coloring the unclear three-dimensional lidar point cloud according to the color information of the preset driving environment region of interest (Region of Interest).

[0011] In a technical solution of the above-mentioned method for processing the three-dimensional point cloud map, the step of "determining a clear three-dimensional laser radar point cloud and an unclear three-dimensional laser radar point cloud according to the laser reflection intensity of each three-dimensional laser radar point cloud in the three-dimensional point cloud map" specifically includes: determining the clear three-dimensional laser radar point cloud and the unclear three-dimensional laser radar point cloud in each frame of the three-dimensional laser radar point cloud of the three-dimensional point cloud map by parallel processing; accordingly, the step of "coloring the unclear three-dimensional laser radar point cloud according to the color information of the two-dimensional image of the driving environment collected by the image acquisition device on the vehicle for visual display" also includes: coloring the unclear three-dimensional laser radar point cloud in each frame of the three-dimensional laser radar point cloud according to the color information of the two-dimensional image of the driving environment collected by the image acquisition device on the vehicle by parallel processing;

[0012] And / or, before the step of "determining clear 3D lidar point clouds and unclear 3D lidar point clouds based on the laser reflection intensity of each 3D lidar point cloud in the 3D point cloud map" or the step of "coloring the unclear 3D lidar point cloud for visual display based on the color information of the 2D image of the driving environment acquired by the image acquisition device on the vehicle", the method also includes: performing ground fitting on the 3D lidar point cloud in the 3D point cloud map to determine the ground plane of the 3D point cloud map; and removing the 3D lidar point clouds belonging to dynamic objects in the 3D point cloud map according to the ground plane.

[0013] In a second aspect, a method for acquiring lane marking data is provided, the method comprising:

[0014] Acquire a three-dimensional laser radar point cloud of a driving environment collected by a laser radar on the vehicle during driving of the vehicle; establish a three-dimensional point cloud map of the driving environment based on the three-dimensional laser radar point cloud; use the three-dimensional point cloud map processing method described in any one of the technical solutions of the above method to colorize unclear three-dimensional laser radar point clouds in the three-dimensional point cloud map; generate a bird's eye view and a height map of the driving environment based on the colored three-dimensional point cloud map; annotate lane elements of the driving environment during driving of the vehicle based on the bird's eye view and the height map to form four-dimensional lane element annotation data containing one-dimensional time information and three-dimensional spatial position information.

[0015] In a technical solution of the above-mentioned method for acquiring lane annotation data, the step of "generating a bird's eye view (Bird Eye View) and a height map (Height Map) of the driving environment according to the three-dimensional point cloud map after coloring" specifically includes: generating a first bird's eye view (Bird Eye View) according to the clear three-dimensional lidar point cloud in the three-dimensional point cloud map after coloring, generating a second bird's eye view (Bird Eye View) according to the unclear three-dimensional lidar point cloud in the three-dimensional point cloud map after coloring, and generating the height map (Height Map) according to the three-dimensional point cloud map after coloring.

[0016] In a technical solution of the above-mentioned method for acquiring lane annotation data, the step of "annotating lane elements of the driving environment during vehicle driving according to the bird's-eye view and the height map to form four-dimensional lane element annotation data including one-dimensional time information and three-dimensional spatial position information" specifically includes: in response to receiving a annotation start instruction, loading and displaying the first bird's-eye view and the height map; in the process of loading the first bird's-eye view and the height map, if a annotation switching instruction is received, loading and displaying the second bird's-eye view and the height map.

[0017] In a third aspect, a computer device is provided, which includes a processor and a storage device, wherein the storage device is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the method for processing a three-dimensional point cloud map or the method for obtaining lane marking data described in any one of the technical solutions of the above method.

[0018] In a fourth aspect, a computer-readable storage medium is provided, which stores a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the method for processing a three-dimensional point cloud map or the method for acquiring lane marking data described in any one of the technical solutions of the method for acquiring lane marking data.

[0019] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects:

[0020] In the technical solution of the present invention, the three-dimensional laser radar point cloud of the driving environment collected by the laser radar on the vehicle during the driving process can be obtained, and then a three-dimensional point cloud map of the driving environment can be established based on the three-dimensional laser radar point cloud; according to the laser reflection intensity of each three-dimensional laser radar point cloud in the three-dimensional point cloud map, the clear three-dimensional laser radar point cloud and the unclear three-dimensional laser radar point cloud are determined; finally, according to the color information of the two-dimensional image of the driving environment collected by the image acquisition device on the vehicle, the unclear three-dimensional laser radar point cloud is colored for visual display. Through the above implementation, the unclear three-dimensional laser radar point cloud in the driving environment can be visualized, and the visualization degree of the three-dimensional point cloud map can be improved, so that the annotation efficiency of lane elements can be improved.

[0021] In another technical solution for implementing the present invention, a three-dimensional laser radar point cloud of the driving environment collected by a laser radar on a vehicle during the driving process of the vehicle can be obtained, and a three-dimensional point cloud map of the driving environment can be established based on the three-dimensional laser radar point cloud; then, the three-dimensional laser radar point cloud that is not clear in the three-dimensional point cloud map is colored using the aforementioned three-dimensional point cloud map processing method, and a bird's eye view (Bird Eye View) and a height map (Height Map) of the driving environment are generated based on the colored three-dimensional point cloud map, and finally, the lane elements of the driving environment during the driving process of the vehicle are annotated based on the bird's eye view and the height map to form four-dimensional lane element annotation data containing one-dimensional time information and three-dimensional spatial position information.

[0022] Through the above implementation, the annotation data of lane elements in four-dimensional space (a four-dimensional space formed by time and three-dimensional space) can be accurately obtained. After using these annotation data to train the vehicle's perception model, the perception ability of the perception model can be greatly improved, so that the real-time local map and obstacle information of the driving environment can be accurately determined during the vehicle's driving, ensuring that the vehicle can drive safely. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. Among them:

[0024] Figure 1 is a schematic flow chart of main steps of a method for processing a three-dimensional point cloud map according to an embodiment of the present invention;

[0025] Figure 2 It is a flowchart diagram of main steps of a method for establishing a three-dimensional point cloud map when the positioning signal of a global satellite positioning device is normal according to an embodiment of the present invention;

[0026] Figure 3 is a flow chart of main steps of a method for removing a three-dimensional laser radar point cloud belonging to a dynamic object according to an embodiment of the present invention;

[0027] Figure 4 is a schematic flow chart of main steps of a method for determining a ground plane of a three-dimensional point cloud map according to an embodiment of the present invention;

[0028] Figure 5 is a schematic diagram of a polar coordinate grid diagram of a three-dimensional point cloud map according to an embodiment of the present invention;

[0029] Figure 6 is a schematic flow chart of main steps of a method for acquiring lane marking data according to an embodiment of the present invention;

[0030] Figure 7 It is a schematic flow chart of the main steps of a method for calibrating the posture parameters for coordinate system conversion between a laser radar and an image acquisition device according to an embodiment of the present invention;

[0031] Figure 8 It is a flowchart diagram of the main steps of a method for calibrating posture parameters for coordinate system conversion between a laser radar and an image acquisition device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0032] Some embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0033] In the description of the present invention, "processor" may include hardware, software or a combination of the two. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware or a combination of the two. Computer-readable storage media include any suitable medium that can store program codes, such as a magnetic disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B or A and B.

[0034] First, an embodiment of a method for processing a three-dimensional point cloud map is described below.

[0035] 1. First Embodiment of a Method for Processing a Three-Dimensional Point Cloud Map

[0036] See attached Figure 1 , Figure 1FIG. 1 is a flow chart showing the main steps of a method for processing a three-dimensional point cloud map according to an embodiment of the present invention. Figure 1 As shown, the method for processing a three-dimensional point cloud map in the embodiment of the present invention mainly includes the following steps S101 to S104.

[0037] Step S101: Acquire a three-dimensional laser radar point cloud of the driving environment collected by a laser radar on the vehicle during the driving process of the vehicle.

[0038] A three-dimensional laser radar point cloud refers to three-dimensional data determined based on the echo signal reflected back to the laser radar (Laser Radar) on the vehicle after the environmental point in the driving environment receives the electromagnetic wave sent to it. The three-dimensional data contains the three-dimensional coordinates of the environmental point in the point cloud coordinate system.

[0039] Step S102: Create a three-dimensional point cloud map of the driving environment based on the three-dimensional laser radar point cloud.

[0040] While the vehicle is driving, the lidar will continuously scan the driving environment to collect three-dimensional lidar point clouds. By splicing each frame of three-dimensional lidar point clouds collected by the lidar in the order of collection, a three-dimensional point cloud map can be obtained.

[0041] Step S103: Determine clear 3D laser radar point clouds and unclear 3D laser radar point clouds according to the laser reflection intensity of each 3D laser radar point cloud in the 3D point cloud map.

[0042] Specifically, the laser reflection intensity can be compared with a preset intensity threshold; if the laser reflection intensity is less than the preset intensity threshold, the 3D laser radar point cloud is an unclear 3D laser radar point cloud; if the laser reflection intensity is greater than or equal to the preset intensity threshold, the 3D laser radar point cloud is a clear 3D laser radar point cloud. Those skilled in the art can flexibly set the specific value of the preset intensity threshold according to actual needs, and this embodiment does not specifically limit this.

[0043] In some preferred embodiments, clear three-dimensional lidar point clouds and unclear three-dimensional lidar point clouds in each frame of three-dimensional lidar point cloud of the three-dimensional point cloud map can be determined separately by parallel processing to improve processing efficiency.

[0044] Step S104: according to the color information of the two-dimensional image of the driving environment acquired by the image acquisition device on the vehicle, the unclear three-dimensional laser radar point cloud is colored for visual display.

[0045] Specifically, the color information of the two-dimensional image can be added to the point cloud information of the unclear three-dimensional laser radar point cloud to achieve the coloring processing of the three-dimensional point cloud map. Similar to step S103, in some preferred embodiments, the unclear three-dimensional laser radar point cloud in each frame of the three-dimensional laser radar point cloud can also be colored by parallel processing according to the color information of the two-dimensional image of the driving environment collected by the image acquisition device on the vehicle to improve the processing efficiency.

[0046] Based on the method described in steps S101 to S104 above, the unclear three-dimensional lidar point cloud in the driving environment can be visualized, thereby improving the visualization of the three-dimensional point cloud map, thereby improving the efficiency of labeling lane elements.

[0047] The above steps S102 and S104 are further described below.

[0048] In order to ensure the accuracy of the three-dimensional point cloud map, in some embodiments of the above step S102, when establishing the three-dimensional point cloud map, for each frame of the three-dimensional laser radar point cloud, the relative position (ego-motion) of the current frame of the three-dimensional laser radar point cloud relative to the three-dimensional point cloud map established based on the previous multiple frames of three-dimensional laser radar point clouds can be determined, and the three-dimensional point cloud map is updated according to the relative position and the current frame of the three-dimensional laser radar point cloud to establish a three-dimensional point cloud map of the driving environment. Among them, for the sake of simplicity, the "three-dimensional point cloud map established based on the previous multiple frames of three-dimensional laser radar point clouds" is referred to as the "prior map".

[0049] For the first frame of 3D laser radar point cloud, there is no prior map, and the first frame of 3D laser radar point cloud can be used as the prior map of the second frame of 3D laser radar point cloud.

[0050] For each non-first frame 3D lidar point cloud, after determining the relative pose of the current frame 3D lidar point cloud and the previous map, point cloud registration can be performed on each 3D lidar point cloud in the current frame 3D lidar point cloud and the 3D lidar point cloud in the previous map according to this relative pose, so as to determine which 3D lidar point cloud in the previous map each 3D lidar point cloud in the current frame 3D lidar point cloud matches or corresponds to, and finally, according to the result of the point cloud registration, the 3D lidar point cloud in the current frame 3D lidar point cloud is added to the previous map, that is, the 3D lidar point cloud of the current frame is spliced ​​with the previous map to form a new map, and this new map becomes the previous map of the next frame 3D lidar point cloud.

[0051] Specifically, in this embodiment, the relative position of the current frame three-dimensional lidar point cloud and the previous map can be determined through the following steps 11 to 16.

[0052] Step 11: Filter out the first edge feature point and the first plane feature point from all the three-dimensional laser radar point clouds contained in the three-dimensional laser radar point cloud of the current frame. Specifically, the roughness of each three-dimensional laser radar point cloud can be calculated, and the three-dimensional laser radar point cloud with a roughness less than a preset roughness threshold is used as the first plane feature point, and the three-dimensional laser radar point cloud with a roughness greater than or equal to the preset roughness threshold is used as the first edge feature point.

[0053] It should be noted that in this embodiment, the conventional point cloud roughness calculation method in the field of three-dimensional point cloud processing technology can be used to calculate the roughness of each three-dimensional laser radar point cloud, and this embodiment does not specifically limit the above calculation method. In addition, those skilled in the art can also flexibly set the specific value of the preset roughness threshold according to actual needs, and this embodiment also does not specifically limit this.

[0054] Step 12: Filter out the second edge feature points and the second plane feature points from the three-dimensional lidar point cloud contained in the previous map, and establish a KD-Tree (K-dimensional search tree) of the three-dimensional lidar point cloud based on the filtered second edge feature points and the second plane feature points.

[0055] The method for selecting the second edge feature points and the second plane feature points from the three-dimensional laser radar point cloud contained in the previous map is the same as the method for selecting the first edge feature points and the first plane feature points in the aforementioned step 11, and will not be repeated here. In addition, in this embodiment, a conventional KD-Tree establishment method can be used to establish a KD-Tree of the three-dimensional laser radar point cloud based on the selected second edge feature points and the second plane feature points.

[0056] Step 13: According to the KD-Tree of the three-dimensional laser radar point cloud, search for the second edge feature point that is the nearest neighbor to the first edge feature point, and search for the second plane feature point that is the nearest neighbor to the first plane feature point.

[0057] Step 14: According to the relative posture, determine the three-dimensional lidar point cloud that matches the first edge feature point on the prior map and the three-dimensional lidar point cloud that matches the first plane feature point, wherein the "three-dimensional lidar point cloud that matches the first edge feature point" is referred to as the "prior matching edge point cloud", and the "three-dimensional lidar point cloud that matches the first plane feature point" is referred to as the "prior matching plane point cloud".

[0058] The previously matched edge point cloud can be expressed as f1(A1,O), where A1 represents the position and posture of the first edge feature point, and O represents the relative position and posture of the current frame three-dimensional lidar point cloud and the previous map.

[0059] The previously matched plane point cloud can be expressed as f2(A2,O), where A2 represents the position and posture of the first plane feature point, and O represents the relative position and posture of the current frame three-dimensional lidar point cloud and the previous map.

[0060] Step 15: Establish an edge feature point loss function based on the previously matched edge point cloud and the second edge feature point, and establish a plane feature point loss function based on the previously matched plane point cloud and the second plane feature point.

[0061] The edge feature point loss function refers to the distance error equation of the line segment formed by the first matching edge point cloud to the second edge feature point. The loss value of the edge feature point loss function is "the distance from the first matching edge point cloud to the line segment formed by the second edge feature point";

[0062] The plane feature point loss function refers to the distance error equation from the first matching plane point cloud to the plane formed by the second plane feature point. The loss value of the plane feature point loss function is "the distance from the first matching plane point cloud to the plane formed by the second plane feature point".

[0063] Step 16: With the loss values ​​of the edge feature point loss function and the plane feature point loss function being less than the preset values ​​as the target, the relative posture O is iteratively optimized, and the relative posture O with the loss values ​​of the edge feature point loss function and the plane feature point loss function being less than the preset values ​​is taken as the final relative posture.

[0064] The above is a detailed description of an implementation of the above step S102. Another implementation of the above step S102 will be described below.

[0065] As the map is continuously updated, the accumulated error will also increase, thereby reducing the accuracy of the three-dimensional point cloud map. In order to overcome the above problem, in some other embodiments of the above step S102, the global satellite positioning device installed on the vehicle can be reused, and the advantage of the high positioning accuracy of the global satellite positioning device can be used to establish a three-dimensional point cloud map using the global satellite positioning device when the positioning signal of the global satellite positioning device is normal to improve the accuracy of the three-dimensional point cloud map, and when the positioning signal of the global satellite positioning device is abnormal, the map is updated according to the relative position of the current frame three-dimensional laser radar point cloud and the previous map to establish a three-dimensional point cloud map.

[0066] A global satellite positioning device refers to a device that uses satellite navigation positioning technology for positioning. In this embodiment, the global satellite positioning device can be a device that performs positioning based on the Global Navigation Satellite System, a device that performs positioning based on the Global Positioning System, a device that performs positioning based on the BeiDou Navigation Satellite System, or a device based on RTK (Real Time Kinematic) positioning technology. Those skilled in the art can flexibly select different types of global satellite positioning devices according to actual needs, and this embodiment does not specifically limit this.

[0067] The following describes the method for establishing a three-dimensional point cloud map when the positioning signal of the global satellite positioning device is normal and abnormal.

[0068] (I) Method for establishing a three-dimensional point cloud map when the positioning signal is normal.

[0069] like Figure 2 As shown, the method for establishing a three-dimensional point cloud map in this embodiment mainly includes the following steps S1021 to S1023.

[0070] Step S1021: Determine the pose parameters T for coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle go In practical applications, after installing a global satellite positioning device and a laser radar on a vehicle, it is necessary to convert the position parameters T between the global satellite positioning device and the laser radar into coordinate systems. go In this embodiment, the position and posture parameters T of the global satellite positioning device can be directly obtained. go The position parameters T of the global satellite positioning device and the laser radar on the vehicle for coordinate system conversion are obtained from the calibration results go .

[0071] Step S1022: Based on the posture parameter T go And according to the global positioning posture obtained by the global satellite positioning device, the radar positioning posture corresponding to each frame of the three-dimensional laser radar point cloud of the driving environment collected by the laser radar is determined. During the driving process of the vehicle, the vibration of the vehicle may cause a large error in the radar positioning posture actually output by the laser radar. If the radar positioning posture actually output by the laser radar is used to establish a three-dimensional point cloud map of the driving environment, it will affect the accuracy of the three-dimensional point cloud map.

[0072] Based on the above posture parameter T goThe determined radar positioning pose is not the radar positioning pose actually output by the laser radar, but the radar positioning pose obtained by coordinate system conversion and matched with the global positioning pose. Since the positioning accuracy of the global satellite positioning device is relatively high, the accuracy of the radar positioning pose matched by the above pose parameters will also be relatively high. Therefore, using the radar positioning pose matched by the above pose parameters to establish a three-dimensional point cloud map of the driving environment will greatly improve the accuracy of the three-dimensional point cloud map.

[0073] Step S1023: stitching each frame of 3D laser radar point cloud according to the radar positioning posture to establish a 3D point cloud map of the driving environment. Specifically, the relative postures of two adjacent frames of 3D laser radar point cloud can be determined according to the radar positioning posture corresponding to each frame of 3D laser radar point cloud, and then point cloud matching is performed on the two frames of 3D laser radar point cloud according to the relative posture, and the two frames of 3D laser radar point cloud are stitched together according to the result of point cloud matching.

[0074] In some embodiments, in order to improve the accuracy of the three-dimensional point cloud map, each frame of the three-dimensional laser radar point cloud may be first subjected to point cloud dedistortion processing, and then the three-dimensional laser radar point cloud after the point cloud dedistortion processing may be spliced ​​to establish a three-dimensional point cloud map of the driving environment. It should be noted that in this embodiment, a conventional point cloud dedistortion method in the field of three-dimensional laser radar point cloud technology may be used to perform point cloud dedistortion processing on each frame of the three-dimensional laser radar point cloud, which will not be described in detail in this embodiment.

[0075] Based on the method described in steps S1021 to S1023 above, the global satellite positioning device can be used to obtain accurate radar positioning posture, and then when the three-dimensional lidar point cloud is spliced ​​according to the radar positioning posture to establish a three-dimensional point cloud map of the driving environment, the accuracy of the three-dimensional point cloud map can be significantly improved.

[0076] The above step S1021 and step S1022 are further explained below.

[0077] First, the above-mentioned step S1021 is explained.

[0078] In order to improve the accuracy of the radar positioning posture obtained by coordinate system conversion based on the posture parameters, in some implementations of the above step S1021, the posture parameter T for coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle can be determined by the following steps 21 to 23.

[0079] Step 21: According to the pose parameters (initial pose parameters of the coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle) and the global positioning pose obtained by the global satellite positioning device when the vehicle turns, determine the radar positioning pose corresponding to each frame of the three-dimensional laser radar point cloud collected by the laser radar when the vehicle turns. Radar positioning pose T o It can be expressed as T o =T go *T g , T go T represents the pose parameters of the coordinate system transformation between the global satellite positioning device and the laser radar on the vehicle, g Represents the global positioning pose obtained by the global satellite positioning device.

[0080] Compared with straight-line driving, the position and posture (position) of the vehicle will change more significantly when turning. Therefore, the position and posture parameters T for coordinate system conversion between the global satellite positioning device and the laser radar on the vehicle can be more accurately determined by using the data when the vehicle turns. go Similar to step S1022, based on the above-mentioned posture parameter T go The determined radar positioning pose is not the radar positioning pose actually output by the lidar, but the radar positioning pose obtained by coordinate system conversion that matches the global positioning pose.

[0081] Step 22: Use the point cloud registration algorithm based on the ICP (Iterative Closest Point) algorithm to calculate the position and posture parameters T according to the corresponding radar positioning posture of each frame of the 3D laser radar point cloud collected by the laser radar when the vehicle turns. go Make optimizations.

[0082] Specifically, a KD-Tree (K-dimensional search tree) of 3D laser radar point clouds is first established; then, based on the KD-Tree, 3D laser radar point clouds with nearest neighbor matching relationships in two adjacent frames of 3D laser radar point clouds are searched; finally, a point cloud registration algorithm based on the ICP (Iterative Closest Point) algorithm is used, and the pose parameters T of the coordinate system transformation between the global satellite positioning device and the laser radar on the vehicle are obtained according to the 3D laser radar point clouds with nearest neighbor matching relationships. go In some preferred implementations, the point cloud registration algorithm based on the ICP algorithm may be a point cloud registration algorithm based on the point-to-plane method in the ICP algorithm. For the sake of brevity, the embodiment of the present invention does not specifically describe the point-to-plane method.

[0083] Step 23: The optimized pose parameters T go As the final pose parameter T go, which is the posture parameter T used in step S1022 go Based on the method described in steps 21 to 23 above, the posture parameter T can be significantly improved. go calibration accuracy.

[0084] The above is a further explanation of step S1021.

[0085] The above step S1022 is further described below.

[0086] In order to accurately obtain the radar positioning posture corresponding to each frame of the three-dimensional lidar point cloud, in some implementations of the above step S1022, the radar positioning posture can be obtained through the following steps 31 to 32.

[0087] Step 31: According to the timestamps corresponding to each frame of the three-dimensional laser radar point cloud, the global positioning pose obtained by the global satellite positioning device is interpolated to obtain the global positioning pose at each timestamp. Since the acquisition time of the global satellite positioning device and the laser radar may not be synchronized, this results in the laser radar collecting a new frame of three-dimensional laser radar point cloud at some moments, but the global satellite positioning device may collect a new frame of positioning signals a little later, that is, the three-dimensional laser radar point cloud actually collected by the laser radar is not one-to-one corresponding to the global positioning pose of the positioning signal collected by the global satellite positioning device. By using the above method of timestamp interpolation calculation for the global positioning pose, the global positioning pose corresponding to each frame of the three-dimensional laser radar point cloud can be accurately obtained, that is, the global positioning pose corresponding to each frame of the three-dimensional laser radar point cloud actually collected by the laser radar is obtained.

[0088] Step 32: Position parameters T of coordinate system conversion based on global satellite positioning device and laser radar go , and according to the global positioning pose at each timestamp, the radar positioning pose corresponding to each frame of the three-dimensional laser radar point cloud is determined respectively. Based on the method described in steps 31 to 32 above, the radar positioning pose corresponding to each frame of the three-dimensional laser radar point cloud actually collected by the laser radar can be accurately obtained.

[0089] The above is a further explanation of step S1022.

[0090] (ii) A method for establishing a three-dimensional point cloud map when the positioning signal is abnormal.

[0091] Similar to the implementation of the aforementioned step S102, when the positioning signal of the global satellite positioning device is abnormal, a three-dimensional point cloud map of the driving environment can be established in the following manner. For each frame of the three-dimensional laser radar point cloud, determine the relative position and posture of the current frame three-dimensional laser radar point cloud relative to the three-dimensional point cloud map established based on the previous multiple frames of three-dimensional laser radar point clouds, and update the three-dimensional point cloud map based on the relative position and posture and the current frame three-dimensional laser radar point cloud to establish a three-dimensional point cloud map of the driving environment. For the sake of simplicity, the "three-dimensional point cloud map established based on the previous multiple frames of three-dimensional laser radar point clouds" is referred to as the "prior map".

[0092] According to step 15 in the aforementioned implementation, when determining the relative pose of the current frame three-dimensional laser radar point cloud with respect to the previous map, the geometric structural features of the point cloud (the distance from the line segment formed by the previously matched edge point cloud to the second edge feature point and the distance from the previously matched plane point cloud to the plane formed by the second plane feature point) are mainly registered to determine the relative pose. However, when the vehicle is traveling in a scene with a strong repeatability of geometric structural features (such as a highway, tunnel or cross-sea bridge, etc.), the above-mentioned method of registering the geometric structural features of the point cloud to determine the relative pose may cause mismatching. In this regard, in order to overcome the above-mentioned problem and improve the robustness of the method for determining the relative pose, in this implementation, the relative pose of the current frame three-dimensional laser radar point cloud with respect to the previous map can be determined by the following steps 41 to 42.

[0093] Step 41: Based on the angular velocity and linear velocity of the vehicle, predict the relative pose of the current frame 3D LiDAR point cloud relative to the 3D point cloud map established based on the previous multiple frames of 3D LiDAR point clouds, and use the predicted relative pose as the prior relative pose.

[0094] Angular velocity refers to the arc turned in a unit time (such as 1 second), and linear velocity refers to the distance traveled in a unit time (such as 1 second). When the positioning signal of the global satellite positioning device is abnormal, the angular velocity and linear velocity from the vehicle's wheel odometer can be received through the CAN (Controller Area Network) bus; when the positioning signal of the global satellite positioning device is normal, the angular velocity and linear velocity can be obtained through the global satellite positioning device.

[0095] The angular velocity and linear velocity of the vehicle usually do not change suddenly within one or several scanning cycles (short time) of the laser radar. Therefore, the posture change of the three-dimensional laser radar point cloud in the current frame can be predicted based on the angular velocity of the vehicle in the scanning cycle corresponding to the previous frame or the previous few frames of the three-dimensional laser radar point cloud. At the same time, the position change of the three-dimensional laser radar point cloud in the current frame can also be predicted based on the linear velocity of the vehicle in the scanning cycle corresponding to the previous frame or the previous few frames of the three-dimensional laser radar point cloud. Based on the above posture change and attitude change, the prediction result of the relative posture of the three-dimensional laser radar point cloud in the current frame relative to the previous map can be obtained.

[0096] Step 42: Based on the prior relative pose, determine the relative pose of the current frame three-dimensional laser radar point cloud relative to the three-dimensional point cloud map established based on the previous multiple frames of three-dimensional laser radar point clouds. Specifically, firstly, the relative pose of the current frame three-dimensional laser radar point cloud relative to the prior map is determined by the method described in steps 11 to 16 in the aforementioned implementation mode. For the sake of simplicity, the relative pose is referred to as the initial relative pose. Then, the initial relative pose is compared with the prior relative pose; if the difference between the two is small, the initial relative pose is used as the final relative pose; if the difference between the two is large, the prior relative pose is used as the final relative pose. Based on the methods described in steps 41 to 42 above, the relative pose of the current frame three-dimensional laser radar point cloud relative to the prior map can be accurately obtained even when the vehicle is driving in a scene with strong repeatability of geometric structure features.

[0097] In addition, in some other embodiments, in order to further improve the robustness of the method for determining relative pose, before determining the relative pose of the current frame three-dimensional lidar point cloud relative to the previous map, dynamic object detection can be performed on the current frame three-dimensional lidar point cloud, and the three-dimensional lidar point cloud belonging to dynamic objects in the current frame three-dimensional lidar point cloud can be removed according to the detection results. Finally, the relative pose is determined based on the current frame three-dimensional lidar point cloud after the three-dimensional lidar point cloud belonging to dynamic objects is removed.

[0098] In this embodiment, the 3D lidar point cloud samples can be used to train the dynamic object detection model established based on the deep convolutional neural network, and then the trained dynamic object detection model is used to perform dynamic object detection on each frame of the 3D lidar point cloud. The dynamic object detection model can output the detection box of each dynamic object in each frame of the 3D lidar point cloud. The 3D lidar point cloud within the detection box is the 3D lidar point cloud belonging to the dynamic object, and these 3D lidar point clouds can be removed.

[0099] Furthermore, in this embodiment, in order to avoid drift of the radar positioning posture and / or the three-dimensional point cloud map corresponding to the current frame three-dimensional lidar point cloud actually output by the lidar, after the three-dimensional point cloud map is updated according to the relative posture of the current frame three-dimensional lidar point cloud with respect to the previous map and the current frame three-dimensional lidar point cloud to establish a three-dimensional point cloud map of the driving environment, the radar positioning posture corresponding to the current frame three-dimensional lidar point cloud can also be optimized through the following step 51.

[0100] Step 51: According to the constraints of the preset factor graph (Pose Graph), the radar positioning pose corresponding to the three-dimensional lidar point cloud of the current frame is optimized using a pose optimization method based on a factor graph (Pose Graph). It should be noted that the pose optimization method based on a factor graph (Pose Graph) is a conventional pose optimization method in the field of positioning and mapping technology. This embodiment does not explain the specific principles of the method, but only explains the constraints used in the method. The constraints of the preset factor graph (Pose Graph) may include radar positioning pose constraints and global positioning pose constraints. The radar positioning pose constraints and global positioning pose constraints are explained below.

[0101] (A) Radar positioning posture constraints.

[0102] In this embodiment, the radar positioning posture constraint condition is as shown in the following formula (1).

[0103]

[0104] The meanings of the parameters in formula (1) are as follows: odom represents the constraint error of the radar positioning pose constraint, represents the absolute pose of the i-1th key frame in the Pose Graph, represents the absolute pose of the i-th keyframe in the factor graph (PoseGraph), Express Perform the inverse operation, represents the relative pose of the i-th key frame relative to the i-1-th key frame, w represents the world coordinate system, and b represents the vehicle coordinate system. The absolute pose refers to the pose relative to the world coordinate system.

[0105] A key frame refers to a frame of 3D LiDAR point cloud whose radar positioning pose changes more than a preset threshold compared to the previous frame of 3D LiDAR point cloud. That is to say, when establishing the factor graph (Pose Graph), it is constructed based on those 3D LiDAR point clouds with large changes in radar positioning pose, and the 3D LiDAR point clouds with small changes in radar positioning pose are removed. Since the 3D LiDAR point clouds with small changes in radar positioning pose have a relatively small impact on vehicle positioning and map updates, and the number of these 3D LiDAR point clouds is also relatively large, removing these 3D LiDAR point clouds with small changes in radar positioning pose can significantly improve the optimization efficiency of pose optimization based on the factor graph (Pose Graph).

[0106] (2) Global positioning posture constraints.

[0107] In this embodiment, the global positioning posture constraint condition is as shown in the following formula (2).

[0108]

[0109] The meanings of the parameters in formula (2) are as follows: rtk represents the constraint error of global positioning pose constraints, represents the global positioning pose at the timestamp corresponding to the jth key frame in the Pose Graph, T rb Indicates the pose parameters for the coordinate system conversion between the vehicle coordinate system and the global satellite positioning device. represents the absolute pose of the jth key frame in the Pose Graph, Express Perform an inverse operation, r represents the coordinate system of the global satellite positioning device, j∈Ω, Ω represents the key frame in the three-dimensional laser radar point cloud collected by the laser radar when the positioning signal of the global satellite positioning device is normal. The constraint goal of the constraint condition of the above preset factor graph (Pose Graph) is to make the constraint error r odom and constraint error r rtk The sum is minimized.

[0110] The above is Figure 1 Detailed description of step S102 of the method embodiment is shown.

[0111] Next, continue Figure 1 Step S104 of the illustrated method embodiment is further described.

[0112] In some implementations of the above step S104, the unclear three-dimensional lidar point cloud can be colored through the following steps S1041 to S1042.

[0113] Step S1041: Determine the nearest neighbor two-dimensional image corresponding to each frame of three-dimensional laser radar point cloud according to the timestamp corresponding to each frame of three-dimensional laser radar point cloud and the timestamp corresponding to each frame of two-dimensional image acquired by the image acquisition device.

[0114] Step S1042: according to the nearest neighbor two-dimensional image, the unclear three-dimensional laser radar point cloud in each frame of three-dimensional laser radar point cloud is colored. Specifically, first, according to the timestamp corresponding to the nearest neighbor two-dimensional image, the radar positioning posture obtained by the laser radar is interpolated and calculated to obtain the radar positioning posture at the timestamp; then, based on the posture parameters of the coordinate system conversion between the laser radar and the image acquisition device, and according to the radar positioning posture at the above timestamp, the image positioning posture corresponding to the nearest neighbor two-dimensional image is determined; then, according to the image positioning posture and the position information of the unclear three-dimensional laser radar point cloud in the radar coordinate system, the projection image point after the unclear three-dimensional laser radar point cloud is projected to the nearest neighbor two-dimensional image is determined; finally, according to the color information of this projection image point, the unclear three-dimensional laser radar point cloud is colored.

[0115] Furthermore, when there are multiple image acquisition devices on the vehicle and the nearest neighbor two-dimensional image corresponding to the current frame three-dimensional laser radar point cloud includes two-dimensional images acquired by multiple image acquisition devices, the two-dimensional image acquired by the last image acquisition device can be selected as the final nearest neighbor two-dimensional image according to the preset arrangement order of the image acquisition devices, and then the unclear three-dimensional laser radar point cloud in the current frame three-dimensional laser radar point cloud is colored according to the final nearest neighbor two-dimensional image. Assume that there are six image acquisition devices A, B, C, D, E and F on the vehicle, and the arrangement order of these six image acquisition devices is ABCDEF. For example, the nearest neighbor two-dimensional image corresponding to the current frame three-dimensional laser radar point cloud includes two-dimensional images a, b, c, d, e and f acquired by the six image acquisition devices A, B, C, D, E and F. Since the last image acquisition device is F, the two-dimensional image f is used as the final nearest neighbor two-dimensional image.

[0116] For another example, the nearest neighbor 2D image corresponding to the current frame 3D laser radar point cloud includes 2D images a, c and e acquired by image acquisition devices A, C and E. Since the last image acquisition device is E, 2D image e is used as the final nearest neighbor 2D image. Based on the method described in steps S1041 to S1042 above, even when multiple image acquisition devices are provided on the vehicle, unclear 3D laser radar point clouds can be accurately and reliably colored.

[0117] In some other implementations of the above step S104, the unclear three-dimensional lidar point cloud can be colored through the following steps S1043 to S1044.

[0118] Step S1043: extracting a preset driving environment region of interest (Region of Interest) in the two-dimensional image. The preset driving environment region of interest (Region of Interest) includes at least the lane area in the driving environment, and does not include the area that may interfere with the point cloud coloring. Among them, the area that may interfere with the point cloud coloring may include at least the image area of ​​the vehicle itself and the interference area caused by image exposure. The interference area caused by image exposure may be the interference area caused by the rolling shutter effect.

[0119] Step S1044: coloring the unclear 3D laser radar point cloud according to the color information of the preset driving environment region of interest. The implementation method of this step is similar to the method described in the aforementioned step S1042, and will not be repeated here.

[0120] So far, the first embodiment of the method for processing a three-dimensional point cloud map has been described in conjunction with the accompanying drawings. Now, the second embodiment of the method for processing a three-dimensional point cloud map will be described.

[0121] 2. Second Embodiment of the Method for Processing 3D Point Cloud Map

[0122] In the method for processing a three-dimensional point cloud map according to the second embodiment of the present invention, the method for processing a three-dimensional point cloud map may include: Figure 1 Steps S101 to S104 of the method embodiment shown in FIG. Figure 1 The main difference between the method embodiment shown is that after executing step S103 and before executing step S104, a step of removing the 3D laser radar point cloud belonging to the dynamic object is also included. Figure 3 , Figure 3 The main process of the method for removing the 3D laser radar point cloud belonging to a dynamic object in an embodiment of the present invention is exemplified. Figure 3 As shown, in this embodiment of the present invention, the three-dimensional laser radar point cloud belonging to the dynamic object can be removed through the following steps S201 to S202.

[0123] Step S201: performing ground fitting on the three-dimensional laser radar point cloud in the three-dimensional point cloud map to determine the ground plane of the three-dimensional point cloud map.

[0124] Step S202: Remove the three-dimensional laser radar point cloud belonging to the dynamic object in the three-dimensional point cloud map according to the ground plane. Specifically, the three-dimensional laser radar point cloud located above the ground plane can be regarded as the three-dimensional laser radar point cloud belonging to the dynamic object. In some embodiments, after determining the ground plane, the three-dimensional laser radar point cloud can be judged whether it is a ground point cloud or a non-ground point cloud according to the ground height at the position corresponding to each three-dimensional laser radar point cloud, and then according to the point cloud height and the ground height corresponding to each three-dimensional laser radar point cloud, and finally all non-ground point clouds are removed. Among them, if the point cloud height is less than or equal to the ground height, the three-dimensional laser radar point cloud is a ground point cloud; otherwise, the three-dimensional laser radar point cloud is a non-ground point cloud. Based on the method described in steps S201 to S202 above, the three-dimensional laser radar point cloud belonging to the dynamic object in the three-dimensional point cloud map can be removed, so that when the lane element is annotated according to the three-dimensional point cloud map, the interference of the three-dimensional laser radar point cloud belonging to the dynamic object on the lane element can be eliminated, thereby improving the annotation efficiency of the lane element.

[0125] The above step S201 is further explained below.

[0126] See attached Figure 4 In some implementations of the above step S201, the ground plane of the three-dimensional point cloud map can be determined by the following steps S2011 to S2014.

[0127] Step S2011: Using a polar coordinate grid representation method based on a concentric zone model, taking the center point of the three-dimensional point cloud map as the pole, a polar coordinate grid diagram of the three-dimensional point cloud map is established. Figure 5 Taking the polar coordinate grid diagram shown in FIG. 1 as an example, the polar coordinate grid representation method based on the concentric zone model is briefly explained.

[0128] Specifically, taking the center point of the three-dimensional point cloud map as the pole, multiple concentric circle areas are formed along the order of the polar diameter values ​​from small to large, and the polar diameter values ​​corresponding to each concentric circle area can be the same or different. Figure 5 As shown, four concentric areas Z1, Z2, Z3 and Z4 are formed in the order of polar diameter values ​​from small to large, and the polar diameter values ​​corresponding to each of them are different. After determining a plurality of concentric areas, each concentric area is gridded again. In the same concentric area, the polar angles corresponding to each grid (the range of polar angles covered by the grid) are the same, and the polar angles corresponding to the grids in different concentric areas may be different. Figure 5As shown, the polar angles corresponding to the grids in the concentric circle areas Z2 and Z3 are the same and the smallest, followed by the concentric circle area Z1 with the smallest polar angle, and the concentric circle area Z4 with the largest polar angle.

[0129] Step S2012: For each grid in the polar coordinate grid map, plane fitting is performed on the three-dimensional laser radar point cloud in the grid to obtain multiple planes in the grid. In this embodiment, a conventional plane fitting method in the field of plane fitting technology can be used to perform plane fitting on the three-dimensional laser radar point cloud in each grid. For example, a RANSAC (RANdom Sample Consensus) algorithm can be used for plane fitting.

[0130] Step S2013: Determine whether the grid is a ground grid based on the plane normal vector of each plane and the height of each three-dimensional lidar point cloud in the grid, and with the constraint that the vertical angle deviation of each plane normal vector is less than a preset angle deviation threshold and the height difference between adjacent three-dimensional lidar point clouds in the grid is less than a preset height difference threshold.

[0131] The vertical angle deviation refers to the angle deviation between the plane normal vector and the vertical direction of the 3D LiDAR point cloud. The vector [0 0 1] can be used to represent vertical upward, or the vector [0 0 -1] can be used to represent vertical downward. After the plane normal vector is determined, the vector angle between the plane normal vector and the above-mentioned vector [0 01] representing vertical upward (or the above-mentioned vector [0 0 -1] representing vertical downward) can be calculated by the vector angle calculation method, and this vector angle is used as the vertical angle deviation.

[0132] If the vertical angle deviation is less than the preset angle deviation threshold, it indicates that the plane normal vector is perpendicular to the ground, and the plane corresponding to the plane normal vector may be the ground plane; if the height difference between adjacent 3D laser radar point clouds in the grid is less than the preset height difference threshold, it indicates that the 3D laser radar point cloud in the grid is relatively smooth, and the grid may also be a ground grid. After satisfying the above two conditions that the vertical angle deviation is less than the preset angle deviation threshold and the height difference between adjacent 3D laser radar point clouds in the grid is less than the preset height difference threshold, the grid can be determined to be a ground grid.

[0133] Step S2014: perform ground fitting on the three-dimensional laser radar point cloud in the ground grid to determine the ground plane of the three-dimensional point cloud map. In this embodiment, a conventional plane fitting method in the field of plane fitting technology can be used to perform ground fitting on the three-dimensional laser radar point cloud in the ground grid to determine the ground plane of the three-dimensional point cloud map, which is not described in detail in this embodiment. Based on the methods described in steps S2011 to S2014 above, the ground plane of the three-dimensional point cloud map can be accurately obtained, thereby improving the accuracy of the ground point cloud and non-ground point cloud, and ultimately improving the accuracy of removing the three-dimensional laser radar point cloud belonging to dynamic objects.

[0134] So far, the second embodiment of the method for processing a three-dimensional point cloud map has been described in conjunction with the accompanying drawings. Now, the embodiment of the method for obtaining lane marking data will be described.

[0135] 1. First Embodiment of the Method for Acquiring Lane Marking Data

[0136] See attached Figure 6 , Figure 6 FIG. 1 is a flow chart showing the main steps of a method for acquiring lane marking data according to an embodiment of the present invention. Figure 6 As shown, the method for acquiring lane marking data in the embodiment of the present invention mainly includes the following steps S301 to S305.

[0137] Step S301: Acquire the three-dimensional laser radar point cloud of the driving environment collected by the laser radar on the vehicle during the driving process. Figure 1 The method described in step S101 in the method embodiment is the same as that described in step S101, and will not be described again here.

[0138] Step S302: Create a three-dimensional point cloud map of the driving environment based on the three-dimensional laser radar point cloud. Figure 1 The method described in step S102 of the method embodiment is the same as that described in step S102 and will not be described again here.

[0139] Step S303: Colorize the unclear 3D laser radar point cloud in the 3D point cloud map. Figure 1 The method of the method embodiment shown performs coloring on unclear three-dimensional lidar point clouds in the three-dimensional point cloud map, which will not be described in detail here.

[0140] Step S304: Generate a bird's eye view and a height map of the driving environment according to the three-dimensional point cloud map after coloring.

[0141] The bird's-eye view refers to an image obtained by projecting the 3D lidar point cloud in the 3D point cloud map onto a plane perpendicular to the height direction of the point cloud, and each image point in the image corresponds one-to-one to each 3D lidar point cloud in the 3D point cloud map.

[0142] Each image point in the height map also corresponds one-to-one to each 3D lidar point cloud in the 3D point cloud map, and each image point stores the point cloud height of its corresponding 3D lidar point cloud.

[0143] It should be noted that those skilled in the art may adopt the conventional bird's-eye view generation method in the field of 3D lidar point cloud processing technology and generate a bird's-eye view of the driving environment based on the 3D lidar point cloud in the 3D point cloud map. In addition, they may also adopt the conventional height map generation method and generate a height map of the driving environment based on the 3D lidar point cloud in the 3D point cloud map. The embodiments of the present invention do not specifically limit the above methods.

[0144] Step S305: Annotate lane elements of the driving environment during vehicle driving according to the bird's-eye view and the height map to form four-dimensional lane element annotation data including one-dimensional time information and three-dimensional spatial position information.

[0145] The three-dimensional point cloud map is established based on the three-dimensional laser radar point cloud during the vehicle's driving process, so this three-dimensional point cloud map is a dynamic map that contains time information. Furthermore, the bird's-eye view and height map determined based on this three-dimensional point cloud map are also dynamic images that contain time information. Lane elements at least include traffic signs on the lane and / or other objects that can serve as markings, among which traffic signs at least include lane lines, stop lines, road signs (such as left turn arrows), traffic lights and traffic signs, etc. Other objects that can serve as markings include at least rod-shaped objects.

[0146] When lane element annotation is performed based on the above-mentioned bird's-eye view and height map, lane element annotation is actually performed based on a sequence of bird's-eye view and a sequence of height map arranged in chronological order, thereby forming four-dimensional lane element annotation data containing one-dimensional time information and three-dimensional spatial position information. Among them, the one-dimensional time information can be determined by the time information of the bird's-eye view or the height map, the two-dimensional plane position in the three-dimensional spatial position information can be determined based on the bird's-eye view, and the height information in the three-dimensional spatial position information can be determined based on the height map.

[0147] Based on the method described in steps S301 to S305 above, the annotation data of lane elements in four-dimensional space (a four-dimensional space formed by time and three-dimensional space) can be accurately obtained. After the vehicle's perception model is trained using these annotation data, the perception ability of the perception model can be greatly improved, so that the real-time local map and obstacle information of the driving environment can be accurately determined during the vehicle's driving, ensuring that the vehicle can drive safely.

[0148] The above steps S304 and S305 are further explained below.

[0149] In some implementations of the above step S304, a first bird's eye view may be generated based on the clear 3D laser radar point cloud in the 3D point cloud map after coloring, a second bird's eye view may be generated based on the unclear 3D laser radar point cloud in the 3D point cloud map after coloring, and a height map may be generated based on the 3D point cloud map after coloring. In the first bird's eye view, each image point corresponds one-to-one to the clear 3D laser radar point cloud; in the second bird's eye view, each image point corresponds one-to-one to the unclear 3D laser radar point cloud.

[0150] Accordingly, in some implementations of the above step S305, in response to receiving a label start instruction, the first bird's eye view and the height map can be loaded and displayed. Meanwhile, during the process of loading the first bird's eye view and the height map, if a label switching instruction is received, the second bird's eye view and the height map can be loaded and displayed.

[0151] Based on the above implementation, the first bird's-eye view (BirdEye View) can be loaded first when starting the lane element labeling work. When some lane elements cannot be seen clearly from the first bird's-eye view (Bird Eye View), the second bird's-eye view (Bird Eye View) can be switched to load to continue completing the lane element labeling work.

[0152] So far, the first embodiment of the method for acquiring lane marking data has been described in conjunction with the accompanying drawings. Now, the second embodiment of the method for acquiring lane marking data will be described.

[0153] 2. Second Embodiment of the Method for Acquiring Lane Marking Data

[0154] In the method for acquiring lane marking data according to the second embodiment of the present invention, the method for acquiring lane marking data may include: Figure 6 Steps S301 to S305 of the method embodiment shown in FIG. Figure 6 The main difference between the embodiment of the method shown is that after executing step S305, it also includes a step of calibrating the posture parameters of the laser radar and the image acquisition device for coordinate system conversion. Figure 7 , Figure 7 The main process of the method for calibrating the posture parameters of the laser radar and the image acquisition device for coordinate system conversion in an embodiment of the present invention is exemplified. Figure 7 As shown, in an embodiment of the present invention, the posture parameters of the laser radar and the image acquisition device for coordinate system conversion can be calibrated through the following steps S401 to S402.

[0155] Step S401: Acquire two-dimensional lane element detection data obtained by performing lane element detection on a two-dimensional image of the driving environment captured by an image acquisition device on the vehicle. In this embodiment, the two-dimensional image samples can be used to train a two-dimensional lane element detection model established based on a deep convolutional neural network, and then the trained two-dimensional lane element detection model is used to perform lane element detection on the two-dimensional image.

[0156] Step S402: Calibrate the pose parameters of the laser radar and the image acquisition device for coordinate system conversion based on the four-dimensional lane element annotation data and the two-dimensional lane element detection data.

[0157] See attached Figure 8 In this embodiment, the posture parameters of the laser radar and the image acquisition device for coordinate system transformation can be calibrated through the following steps S4021 to S4024.

[0158] Step S4021: Project the four-dimensional lane element annotation data to the coordinate system of the image acquisition device according to the pose parameters to obtain the projection point f(A,O) of the four-dimensional lane element annotation data, where A represents the coordinate of the four-dimensional lane element annotation data in the coordinate system of the laser radar, and O represents the pose parameters.

[0159] Step S4022: Determine the two image points on the two-dimensional image that are closest to the projection point f(A, O).

[0160] Step S4023: According to the line segment cd formed by the projection point f(A, O) and the two image points, a distance error equation from the projection point f(A, O) to the line segment cd is established.

[0161] The distance error equation from the projection point f(A,O) to the line segment cd is shown in equation (3).

[0162] loss=d(f(A,O),cd) (3)

[0163] The meanings of the parameters in the above formula (3) are as follows:

[0164] d represents the distance calculation function from the projection point f(A,O) to the line segment cd, and loss represents the distance calculated by the distance calculation function d.

[0165] Step S4024: With the goal of making the distance loss less than a preset distance threshold, iteratively optimize the posture parameter O in the distance error equation, and obtain the posture parameter O when the distance loss is less than the preset distance threshold, and use the posture parameter O as the calibrated posture parameter.

[0166] In the embodiment of the present invention, the least squares algorithm can be used to iteratively optimize the posture parameter O in the distance error equation. For example, the Levenberg-Marquardt algorithm in the least squares algorithm can be used to iteratively optimize the posture parameter O in the distance error equation. The embodiment of the present invention does not specifically limit the above iterative optimization method.

[0167] Based on the methods described in the above steps S4021 to S4024, the accuracy of calibrating the posture parameters of the laser radar and the image acquisition device for coordinate system conversion can be improved.

[0168] Further, in the method for acquiring lane annotation data according to the second embodiment of the present invention, in addition to including the aforementioned steps S401 to S402, the method for acquiring lane annotation data can also determine the lane element annotation data on the two-dimensional image of the driving environment based on the calibrated posture parameters and according to the four-dimensional lane element annotation data.

[0169] Specifically, the coordinate system of the four-dimensional lane element annotation data can be transformed based on the calibrated posture parameters to determine the image points corresponding to the four-dimensional lane element annotation data on the two-dimensional image, and the lane element annotation data on the two-dimensional image can be determined based on these image points. In this way, the efficiency and accuracy of obtaining the lane element annotation data on the two-dimensional image can be greatly improved.

[0170] The above is a description of the second embodiment of the method for acquiring lane marking data.

[0171] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0172] It is understood by those skilled in the art that the present invention implements all or part of the processes in the method of the above embodiment, and can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0173] Furthermore, the present invention also provides a computer device.

[0174] In an embodiment of a computer device according to the present invention, the computer device includes a processor and a storage device, the storage device can be configured to store a program for executing the method for processing a three-dimensional point cloud map or a method for obtaining lane marking data of the above method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to the program for executing the method for processing a three-dimensional point cloud map or a method for obtaining lane marking data of the above method embodiment. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer device can be a control device device formed by various electronic devices.

[0175] Furthermore, the present invention also provides a computer-readable storage medium.

[0176] In an embodiment of a computer-readable storage medium according to the present invention, the computer-readable storage medium may be configured to store a program for executing the method for processing a three-dimensional point cloud map or the method for obtaining lane marking data of the above-mentioned method embodiment, and the program may be loaded and run by a processor to implement the above-mentioned method for processing a three-dimensional point cloud map or the method for obtaining lane marking data. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium may be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-temporary computer-readable storage medium.

[0177] So far, the technical solution of the present invention has been described in conjunction with an embodiment shown in the accompanying drawings, but it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A method for acquiring lane marking data, characterized in that: The method comprises: Acquire a three-dimensional laser radar point cloud of a driving environment collected by a laser radar on the vehicle during driving of the vehicle; Establishing a three-dimensional point cloud map of the driving environment according to the three-dimensional laser radar point cloud; Determining a clear 3D laser radar point cloud and an unclear 3D laser radar point cloud according to the laser reflection intensity of each 3D laser radar point cloud in the 3D point cloud map; According to the color information of the two-dimensional image of the driving environment acquired by the image acquisition device on the vehicle, the unclear three-dimensional laser radar point cloud is colored for visual display; Generating a bird's eye view and a height map of the driving environment respectively according to the three-dimensional point cloud map after coloring; Annotating lane elements of the driving environment during vehicle driving according to the bird's eye view and the height map to form four-dimensional lane element annotation data including one-dimensional time information and three-dimensional spatial position information; in, The one-dimensional time information is determined by the time information of the bird's-eye view or the height map; The two-dimensional plane position in the three-dimensional spatial position information is determined according to the bird's-eye view, and the height information in the three-dimensional spatial position information is determined according to the height map; The method includes: generating a first bird's eye view (Bird Eye View) and a second bird's eye view (Bird Eye View) respectively according to the clear three-dimensional lidar point cloud and the unclear three-dimensional lidar point cloud in the three-dimensional point cloud map after the coloring process, wherein the first bird's eye view is used to be loaded first when marking the lane elements, and the second bird's eye view is used to be loaded when the lane elements cannot be clearly seen from the first bird's eye view.

2. The method for acquiring lane marking data according to claim 1, characterized in that: The step of "coloring the unclear three-dimensional laser radar point cloud according to the color information of the two-dimensional image of the driving environment collected by the image acquisition device on the vehicle for visual display" specifically includes: Determine the nearest neighbor two-dimensional image corresponding to each frame of the three-dimensional laser radar point cloud according to the timestamp corresponding to each frame of the three-dimensional laser radar point cloud and the timestamp corresponding to each frame of the two-dimensional image collected by the image acquisition device; According to the nearest neighbor two-dimensional image, the unclear three-dimensional laser radar point cloud in each frame of the three-dimensional laser radar point cloud is colored.

3. The method for acquiring lane marking data according to claim 2, characterized in that: The number of image acquisition devices on the vehicle is multiple, and the step of "coloring unclear three-dimensional laser radar point clouds in each frame of three-dimensional laser radar point cloud according to the nearest neighbor two-dimensional image" specifically includes: For each frame of the three-dimensional laser radar point cloud, if the nearest neighbor two-dimensional image corresponding to the current frame of the three-dimensional laser radar point cloud includes two-dimensional images acquired by multiple image acquisition devices, then according to a preset arrangement order of the image acquisition devices, the last two-dimensional image acquired by the image acquisition device is selected as the final nearest neighbor two-dimensional image; The unclear three-dimensional laser radar point cloud in the current frame three-dimensional laser radar point cloud is colored according to the final nearest neighbor two-dimensional image.

4. The method for acquiring lane marking data according to claim 1, characterized in that: The step of "coloring the unclear three-dimensional laser radar point cloud according to the color information of the two-dimensional image of the driving environment collected by the image acquisition device on the vehicle for visual display" also includes: Extracting a preset driving environment region of interest (Region of Interest) in the two-dimensional image; The unclear three-dimensional laser radar point cloud is colored according to the color information of the preset driving environment region of interest.

5. The method for acquiring lane marking data according to claim 1, characterized in that: The step of “determining a clear three-dimensional laser radar point cloud and an unclear three-dimensional laser radar point cloud according to the laser reflection intensity of each three-dimensional laser radar point cloud in the three-dimensional point cloud map” specifically includes: By means of parallel processing, respectively determining clear 3D laser radar point clouds and unclear 3D laser radar point clouds in each frame of the 3D laser radar point cloud of the 3D point cloud map; Accordingly, the step of "coloring the unclear three-dimensional laser radar point cloud according to the color information of the two-dimensional image of the driving environment acquired by the image acquisition device on the vehicle for visual display" also includes: By means of parallel processing, the unclear three-dimensional laser radar point cloud in each frame of the three-dimensional laser radar point cloud is respectively colored according to the color information of the two-dimensional image of the driving environment collected by the image acquisition device on the vehicle; and / or, Before the step of "determining a clear 3D laser radar point cloud and an unclear 3D laser radar point cloud according to the laser reflection intensity of each 3D laser radar point cloud in the 3D point cloud map" or the step of "coloring the unclear 3D laser radar point cloud according to the color information of the 2D image of the driving environment acquired by the image acquisition device on the vehicle for visual display", the method further includes: Performing ground fitting on the three-dimensional laser radar point cloud in the three-dimensional point cloud map to determine the ground plane of the three-dimensional point cloud map; The three-dimensional laser radar point cloud belonging to the dynamic object in the three-dimensional point cloud map is removed according to the ground plane.

6. The method for acquiring lane marking data according to claim 1, characterized in that: The step of "marking the lane elements of the driving environment during the vehicle driving process according to the bird's eye view and the height map to form four-dimensional lane element marking data including one-dimensional time information and three-dimensional spatial position information" specifically includes: In response to receiving the marking start instruction, loading and displaying the first bird's eye view and the height map; During the process of loading the first bird's eye view and the height map, if a label switching instruction is received, the second bird's eye view and the height map are loaded and displayed.

7. A computer device comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by the processor to execute the method for acquiring lane marking data according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the method for acquiring lane marking data according to any one of claims 1 to 6.

Images