A laser radar calibration method, computer device, readable storage medium and motor vehicle

Abstract

The application discloses a laser radar calibration method, computer equipment, a readable storage medium and a motor vehicle, and relates to the technical field of automatic driving, and comprises the following steps: a laser radar acquires true value data of a measured target; a vision sensor acquires a surround view image; the true value data of the measured target is projected on the corresponding surround view image to form a 3D detection frame; a corresponding radar 2D detection frame is formed based on the 3D detection frame; target detection is performed on the surround view image to form a corresponding vision 2D detection frame; an intersection-over-union of the radar 2D detection frame and the vision 2D detection frame is calculated; and the radar 2D detection frame and the vision 2D detection frame are matched based on the intersection-over-union to calibrate the laser radar. The laser radar calibration method provided by the application can correct false detection and missed detection and the like in true value data obtained by a laser radar algorithm, and improves the accuracy of the true value data.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a lidar calibration method, a computer device, a readable storage medium, and a motor vehicle. Background Technology

[0002] With the continuous advancement of vehicle technology, autonomous driving technology has gradually become a hot topic, attracting widespread attention from major automakers. Environmental perception, as the primary component of an autonomous driving system, is a crucial basis for vehicle behavior decisions and path planning. Environmental perception algorithms include vision-based and radar-based algorithms. LiDAR-based perception algorithms are generally used for 3D target detection, capable of obtaining information on the position, size, posture, and category of surrounding vehicles, pedestrians, and other targets in the driving environment. Compared to pure vision-based perception algorithms, LiDAR-based algorithms can generally control the error in detecting the position, size, and posture of targets to the centimeter level, far exceeding pure vision algorithms. However, on the other hand, pure vision-based perception algorithms possess rich semantic information, enabling them to obtain more accurate object category information. Currently, the development of autonomous driving technology typically involves data acquisition and manual annotation teams. However, the process of "manual data acquisition followed by manual annotation" requires a significant amount of time to accumulate ground truth data for algorithm training. In addition to manual annotation, it is also necessary to purchase supplier data or build a ground truth data acquisition vehicle in-house, and coordinate with ground truth data algorithms to obtain the necessary target information, consuming substantial human, material, and financial resources. In addition, the true value data of target objects obtained based on the LiDAR perception algorithm inevitably contains false detections and missed detections. These erroneous data introduced by the algorithm can cause serious errors in other algorithms or functions developed based on it. Summary of the Invention

[0003] To address the aforementioned problems, this invention provides a lidar calibration method that corrects for false detections and missed detections in the ground truth data obtained by lidar algorithms, thereby improving the accuracy of the ground truth data.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] A lidar calibration method includes the following steps:

[0006] The lidar acquires ground truth data of the measured targets around the vehicle in the world coordinate system; the vision sensor acquires a surround view image of the vehicle's surroundings.

[0007] The true value data of the target objects around the vehicle are projected onto the corresponding surround view image to form a 3D detection box;

[0008] The corresponding 2D radar detection frame is formed based on the 3D detection frame;

[0009] Target detection is performed on the panoramic image to generate corresponding visual 2D bounding boxes;

[0010] Calculate the intersection-union ratio (IUGR) of the radar 2D detection boxes and the visual 2D detection boxes;

[0011] The LiDAR is calibrated by matching the radar 2D detection box and the visual 2D detection box based on cross-union comparison.

[0012] Optionally, forming the corresponding 2D radar detection box based on the 3D detection box includes the following steps:

[0013] Obtain the pixel coordinates of the 8 corner points of the 3D detection box projected onto the corresponding panoramic image. Take the maximum and minimum values ​​of the U-axis and V-axis of the 8 corner points in the image coordinate system to generate two new corner points. Generate the corresponding radar 2D detection box from the two newly generated corner points.

[0014] Optionally, matching radar 2D detection boxes and visual 2D detection boxes based on intersection-union comparison includes the following steps:

[0015] Calculate the cross-union ratio (CUP) between all radar 2D bounding boxes and all visual 2D bounding boxes;

[0016] Construct a cost matrix based on the calculated intersection-union ratio;

[0017] In the cost matrix, the optimal match is the one where the sum of the overall costs is maximized.

[0018] Optionally, the results of matching radar 2D detection boxes and visual 2D detection boxes include:

[0019] If the intersection-union ratio is greater than 0 and the categories match each other, the ground truth data of the corresponding target is retained.

[0020] If the radar 2D detection box does not match the visual 2D detection box, the ground truth data of the corresponding target will be removed.

[0021] If the visual 2D detection box does not match the radar 2D detection box, determine whether the corresponding target exists. If it exists, supplement the ground truth data of the target. If it does not exist, remove the visual 2D detection box.

[0022] Optionally, determining whether the corresponding target exists includes the following steps:

[0023] The point cloud data around the vehicle is projected onto the corresponding surround view image, and all point clouds falling within the visual 2D detection box are recorded. Then, the point clouds falling within the visual 2D detection box are clustered in the point cloud space. If the clustering is successful and the clustering result matches the height of the preset category, the target is considered to exist. If the clustering fails or the height of the clustered 3D detection box does not match the preset category, the target is considered not to exist.

[0024] The technical solution provided by this invention introduces a vision-based target detection algorithm to process images acquired by a vehicle-mounted surround-view camera synchronized with the LiDAR in time and space, thereby obtaining information on target objects around the vehicle. This expands the applicability of the LiDAR calibration method. At the same time, it verifies the true data obtained by the LiDAR, eliminates falsely detected targets, and supplements missed targets, thus completing the correction of the true data. This is equivalent to introducing a camera on the hardware basis of the LiDAR, increasing the source of true data, improving the efficiency of data correction, and saving manpower, material resources, and financial resources.

[0025] Meanwhile, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the lidar calibration method described in any of the preceding claims.

[0026] Furthermore, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the lidar calibration method described in any of the preceding claims.

[0027] Furthermore, the present invention also provides a motor vehicle having an autonomous driving function and equipped with a lidar, a vision sensor, and an MCU. When the motor vehicle is operating the autonomous driving function, the lidar is used to identify the surroundings of the motor vehicle operating the autonomous driving function. When calibrating the autonomous driving function, the lidar is calibrated using the MCU and the vision sensor through any of the aforementioned lidar calibration methods.

[0028] Or the motor vehicle has the aforementioned computer equipment;

[0029] Alternatively, the motor vehicle may have the aforementioned computer-readable storage medium, and the computer program, when executed by a processor, implements the lidar calibration method described in any of the preceding claims.

[0030] These features and advantages of the present invention will be disclosed in detail in the following specific embodiments and accompanying drawings. The preferred embodiments or means of the present invention will be shown in detail in conjunction with the accompanying drawings, but are not intended to limit the technical solutions of the present invention. In addition, each of these features, elements and components appearing in the following text and drawings is a plurality of, and different symbols or numbers are used for convenience of representation, but all represent parts with the same or similar construction or function. Attached Figure Description

[0031] The present invention will be further described below with reference to the accompanying drawings:

[0032] Figure 1 This is a flowchart of an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram of the intersection-over-union ratio between the radar 2D detection box and the visual 2D detection box in an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram illustrating the matching of radar 2D detection boxes and visual 2D detection boxes in an embodiment of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be explained and described below with reference to the accompanying drawings. However, the following embodiments are only preferred embodiments of the present invention and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments in the implementation methods without creative effort are all within the protection scope of the present invention.

[0036] The terms "an embodiment," "example," or "trademark" used in this specification refer to a particular feature, structure, or characteristic described in connection with the embodiment itself that may be included in at least one embodiment disclosed in this patent. The phrase "in an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0037] Example:

[0038] When ground truth data contains falsely detected or missed targets, although improving the LiDAR perception algorithm can optimize detection performance, it cannot overcome the inherent limitations of the LiDAR sensor itself, namely the lack of semantic information. Furthermore, the point cloud distribution is characterized by high density at close range and sparseness at distant ranges, which also limits the performance of the LiDAR perception algorithm due to hardware constraints. Therefore, as... Figure 1 As shown, this embodiment provides a lidar calibration method, including the following steps:

[0039] The lidar acquires ground truth data of the measured targets around the vehicle in the world coordinate system; the vision sensor acquires surround view images around the vehicle. Specifically, in this embodiment, the surround view images acquired by the vision sensor are the six-channel surround view images commonly used in the art, namely, the front left image, the left side image, the rear left image, the front right image, the right side image, and the rear right image.

[0040] The true value data of the target objects around the vehicle are projected onto the corresponding surround view image to form a 3D detection box;

[0041] The corresponding 2D radar detection box is generated based on the 3D detection box. This step includes the following sub-steps:

[0042] Since the 3D detection box is cubic in shape and has 8 corner points, the pixel coordinates of the 8 corner points projected onto the corresponding panoramic image are obtained, namely [U0, V0], [U1, V1]...[U7, V7]. The maximum and minimum values ​​of the 8 corner points on the U-axis and V-axis in the image coordinate system are then taken, i.e., [U0, V0], [U1, V1]...[U7, V7]. min V min ], [U max V max Two new corner points are generated, and the corresponding radar 2D detection boxes are generated from the two newly generated corner points.

[0043] The panoramic image is subjected to object detection to form corresponding visual 2D detection boxes. In this step, the object detection algorithm based on visual perception used can be a mature object detection algorithm with high accuracy that is already available in the technology. This is existing technology in this field and is not limited here.

[0044] Calculate the intersection-union ratio (IoU) of the radar 2D detection boxes and the visual 2D detection boxes, such as... Figure 2 As shown, the crossover and union ratio

[0045] The LiDAR is calibrated by matching the radar 2D detection boxes and the visual 2D detection boxes based on the intersection-union comparison (CIUC). Specifically, the matching of the radar 2D detection boxes and the visual 2D detection boxes based on CIUC uses Hungarian matching. If no augmenting path is found starting from a certain point, then no matter how many augmenting paths are found from other points to change the current matching, no augmenting path will ever be found starting from that point. Therefore, the maximum matching is empty initially, and then the process of searching for and expanding augmenting paths is continuously repeated until no augmenting path can be found. In this embodiment, the matching includes the following steps, such as... Figure 3 As shown:

[0046] Calculate the cross-union ratio (CUP) between all radar 2D bounding boxes and all visual 2D bounding boxes;

[0047] Construct a cost matrix based on the calculated intersection-union ratio;

[0048] In the cost matrix, the optimal match is the one where the sum of the overall costs is maximized. For example, suppose the target detection algorithm detects 3 targets, named V1, V2, and V3, and the LiDAR algorithm detects 4 targets, named L1, L2, L3, and L4. Calculate all IOU values ​​for matching the two algorithms and construct the cost matrix. The rows can be either those for visually detected targets or those for LiDAR-detected targets; there is no limitation here. The optimal match can be found by finding the one with the minimum or maximum sum of overall costs, as shown in the table below:

[0049]

[0050] The matching result is:

[0051] In this embodiment, the matching results are: V1 matches L2, V2 matches L3, and V3 matches L4.

[0052] The matching results include:

[0053] If the intersection-union ratio is greater than 0 and they match each other, that is, the preset categories are consistent, then the true value data of the corresponding target being tested is retained;

[0054] If the radar 2D detection box does not match the visual 2D detection box, it is considered a false detection, and the ground truth data of the corresponding target will be removed.

[0055] If the visual 2D detection box does not match the radar 2D detection box, there are two possibilities: one is that the target is a false detection by the 2D vision algorithm, and the other is that the target is a missed detection by the LiDAR algorithm. Therefore, it is necessary to determine whether the corresponding target exists according to the following steps;

[0056] The point cloud data around the vehicle is projected onto the corresponding surround view image, and all point clouds falling within the visual 2D detection boxes are recorded. Then, the point clouds falling within the visual 2D detection boxes are clustered in the point cloud space. The clustering results are represented by radar 3D detection boxes, which have the center point position, length, width, and height. If the clustering is successful and the clustering result matches the height of the preset category, the target is considered to exist, and the corresponding ground truth data of the target is supplemented. If the clustering fails or the height of the clustered 3D detection boxes does not match the preset category, the target is considered not to exist, and the visual 2D detection boxes are removed.

[0057] The technical solution provided in this embodiment introduces a vision-based target detection algorithm to process images acquired by a vehicle-mounted surround-view camera synchronized with the LiDAR in time and space, thereby obtaining information on target objects around the vehicle. This expands the applicability of the LiDAR calibration method. At the same time, it verifies the true data obtained by the LiDAR, removes falsely detected targets, and supplements missed targets, thus completing the correction of the true data. This is equivalent to introducing a camera on the hardware basis of the LiDAR, increasing the source of dependence for true data, improving the efficiency of data correction, and saving manpower, material resources, and financial resources.

[0058] Meanwhile, this embodiment also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the point cloud semantic segmentation method described above.

[0059] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can implement the methods of any of the above embodiments. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0060] Furthermore, this embodiment also provides a motor vehicle with an autonomous driving function, equipped with a lidar, a vision sensor, and an MCU. When the motor vehicle in this embodiment operates the autonomous driving function, the lidar is used to identify the surroundings of the motor vehicle operating the autonomous driving function. When calibrating the autonomous driving function, the lidar is calibrated using the MCU and the vision sensor, through the previous lidar calibration method.

[0061] Or the motor vehicle has the aforementioned computer equipment;

[0062] The vehicle or motor vehicle has the aforementioned computer-readable storage medium, and the computer program, when executed by a processor, implements the lidar calibration method described in any of the preceding claims.

[0063] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes, but is not limited to, the contents described in the accompanying drawings and the specific embodiments above. Any modifications that do not depart from the functional and structural principles of the present invention will be included within the scope of the claims.

Claims

1. A lidar calibration method, characterized in that, Includes the following steps: The lidar acquires ground truth data of the measured targets around the vehicle in the world coordinate system; the vision sensor acquires a surround view image of the vehicle's surroundings. The true value data of the target objects around the vehicle are projected onto the corresponding surround view image to form a 3D detection box; The corresponding 2D radar detection frame is formed based on the 3D detection frame; Target detection is performed on the panoramic image to generate corresponding visual 2D bounding boxes; Calculate the intersection-union ratio (IUGR) of the radar 2D detection boxes and the visual 2D detection boxes; The LiDAR is calibrated by matching the radar 2D detection box and the visual 2D detection box based on the cross-union ratio. The matching of radar 2D detection boxes and visual 2D detection boxes based on cross-union comparison includes the following steps: Calculate the cross-union ratio (CUP) between all radar 2D bounding boxes and all visual 2D bounding boxes; Construct a cost matrix based on the calculated intersection-union ratio; In the cost matrix, the optimal match is the one where the sum of the overall costs is maximized. The results of matching radar 2D detection boxes and visual 2D detection boxes include: If the intersection-union ratio is greater than 0 and the categories match each other, the ground truth data of the corresponding target is retained. If the radar 2D detection box does not match the visual 2D detection box, the ground truth data of the corresponding target will be removed. If the visual 2D detection box does not match the radar 2D detection box, determine whether the corresponding target exists. If it exists, supplement the ground truth data of the target. If it does not exist, remove the visual 2D detection box.

2. The lidar calibration method according to claim 1, characterized in that, The process of generating a corresponding 2D radar detection box based on a 3D detection box includes the following steps: Obtain the pixel coordinates of the 8 corner points of the 3D detection box projected onto the corresponding panoramic image. Take the maximum and minimum values ​​of the U-axis and V-axis of the 8 corner points in the image coordinate system to generate two new corner points. Generate the corresponding radar 2D detection box from the two newly generated corner points.

3. The lidar calibration method according to claim 1, characterized in that, Determining whether the corresponding target exists involves the following steps: The point cloud data around the vehicle is projected onto the corresponding surround view image, and all point clouds falling within the visual 2D detection box are recorded. Then, the point clouds falling within the visual 2D detection box are clustered in the point cloud space. If the clustering is successful and the clustering result matches the height of the preset category, the target is considered to exist. If the clustering fails or the height of the clustered 3D detection box does not match the preset category, the target is considered not to exist.

4. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the lidar calibration method according to any one of claims 1 to 3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the lidar calibration method according to any one of claims 1 to 3.

6. A motor vehicle, characterized in that, The motor vehicle has an autonomous driving function and is equipped with a lidar, a vision sensor and an MCU. When the motor vehicle is operating the autonomous driving function, the lidar is used to identify the surroundings of the motor vehicle operating the autonomous driving function. When calibrating the autonomous driving function, the lidar is calibrated using the MCU and the vision sensor through the lidar calibration method described in any one of claims 1 to 3. Or the motor vehicle may have the computer equipment as described in claim 4; Or the motor vehicle may have the computer-readable storage medium of claim 5.

Images