Online sensor calibration methods, devices, and storage media for autonomous vehicles

Abstract

This application discloses an online calibration method and apparatus for sensors of autonomous vehicles. The method includes: determining whether the driving lane of the autonomous vehicle meets online calibration conditions; when the online calibration conditions are met, acquiring road image data collected by the autonomous vehicle's camera and laser point cloud data collected by the LiDAR, performing target detection on the road image data and laser point cloud data to obtain image-detected targets and point cloud-detected targets; determining whether the image-detected targets and point cloud-detected targets are the same target; when they are the same target, acquiring first feature point data of the image-detected target and second feature point data of the point cloud-detected target, performing matching calculations on the first feature point data and second feature point data to obtain the online calibration result between the camera and the LiDAR. The technical solution of this application can quickly and accurately complete the online calibration of sensors of autonomous vehicles.

Description

Technical Field

[0001] This application relates to the field of sensor calibration technology, and in particular to an online sensor calibration method and apparatus for autonomous vehicles. Background Technology

[0002] Multi-sensor fusion technology is a crucial algorithm module in the field of autonomous driving, but it heavily relies on the synchronization of time and space. Time synchronization can often be easily solved by hardware triggering, but there is no good solution for spatial synchronization.

[0003] Spatial synchronization manifests as the calibration of extrinsic parameters between various sensors. Existing calibration methods are mostly offline, such as calibration workshops or manual calibration, to calibrate the initial extrinsic parameters. However, during the driving of an autonomous vehicle or after a period of time, the relative poses between sensors change due to vibration and other factors.

[0004] To address the aforementioned issues, some online calibration methods exist in the existing technology. However, most of these methods are based on the matching of linear features between image data and point cloud data, and existing matching algorithms struggle to obtain efficient and accurate online calibration results. Summary of the Invention

[0005] In view of the above-mentioned problems in the prior art, this application provides a method and apparatus for online calibration of sensors in autonomous vehicles to quickly and accurately calculate the relative extrinsic parameters between sensors.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide an online sensor calibration method for an autonomous vehicle, the method comprising:

[0008] Determine whether the driving lane of the autonomous vehicle meets the online calibration conditions;

[0009] When the online calibration conditions are met, road image data collected by the camera of the autonomous vehicle and laser point cloud data collected by the lidar are acquired. Target detection is performed on the road image data and laser point cloud data to obtain image detection targets and point cloud detection targets.

[0010] Determine whether the image detection target and the point cloud detection target are the same target;

[0011] When the targets are the same, the first feature point data of the image detection target and the second feature point data of the point cloud detection target are obtained. The first feature point data and the second feature point data are matched and calculated to obtain the online calibration result between the camera and the lidar.

[0012] Optionally, determining whether the driving lane of the autonomous vehicle meets the online calibration conditions includes:

[0013] Obtain the lane type of the current lane being driven by the autonomous vehicle;

[0014] If the autonomous vehicle is currently traveling in a straight lane, the online calibration conditions are met; otherwise, the online calibration conditions are not met.

[0015] Optionally, determining whether the image detection target and the point cloud detection target are the same target includes:

[0016] Determine whether the image detection target and the point cloud detection target are within the lane currently being traveled by the autonomous vehicle;

[0017] If the target is in the lane where the autonomous vehicle is currently traveling, then determine whether the image detection target and the point cloud detection target are both targets in front of the vehicle or both targets behind the vehicle.

[0018] If both are targets in front of the vehicle or both are targets behind the vehicle, then the image detection target and the point cloud detection target are determined to be the same target.

[0019] Optionally, determining whether the image detection target and the point cloud detection target are within the lane currently being traveled by the autonomous vehicle includes:

[0020] Lane line detection is performed on the road image data to obtain the lane boundary line of the lane currently being traveled by the autonomous vehicle;

[0021] The positional relationship between the image detection target and the lane boundary line is obtained. If the image detection target is within the lane boundary line, it is determined that the image detection target is within the lane currently being traveled by the autonomous vehicle.

[0022] In addition, the lane boundary line of the current driving lane of the autonomous vehicle is virtually generated based on the vehicle width information of the autonomous vehicle in the laser point cloud data.

[0023] The positional relationship between the point cloud detection target and the virtual lane boundary line is obtained. If the point cloud detection target is within the virtual lane boundary line, it is determined that the point cloud detection target is within the lane currently being traveled by the autonomous vehicle.

[0024] Optionally, determining whether the image detection target and the point cloud detection target are both targets in front of the vehicle or both targets behind the vehicle includes:

[0025] The positional relationship between the image detection target and the autonomous vehicle, and the positional relationship between the point cloud detection target and the autonomous vehicle are obtained.

[0026] If the image detection target and the point cloud detection target are located in front of the autonomous vehicle, then the image detection target and the point cloud detection target are both determined to be targets in front of the vehicle.

[0027] If the image detection target and the point cloud detection target are located behind the autonomous vehicle, then the image detection target and the point cloud detection target are both determined to be targets behind the vehicle.

[0028] Optionally, the method further includes:

[0029] If there are multiple image detection targets and point cloud detection targets that are both in front of the vehicle and are located in the lane where the autonomous vehicle is currently traveling, then the image detection target and point cloud detection target that is closest to the autonomous vehicle is determined as the optimal target;

[0030] The process of acquiring the first feature point data of the image detection target and the second feature point data of the point cloud detection target includes:

[0031] Obtain the optimal first feature point data of the image detection target and the optimal second feature point data of the point cloud detection target.

[0032] Optionally, the image detection target includes an image target detection box, and the point cloud detection target includes a point cloud target detection box. The step of acquiring first feature point data of the image detection target and second feature point data of the point cloud detection target, and performing matching calculations on the first and second feature point data to obtain the online calibration result between the camera and the lidar includes:

[0033] The corner data of the target detection box in the image is obtained as the first feature point data, and the corner data of the target detection box in the point cloud is obtained as the second feature point data.

[0034] The relative extrinsic parameters between the camera and the lidar are obtained by using the multi-point perspective imaging (PNP) algorithm on the first feature point data and the second feature point data.

[0035] The online calibration result is obtained based on the relative extrinsic parameters between the camera and the lidar.

[0036] Secondly, embodiments of this application also provide an online sensor calibration device for autonomous vehicles, the device comprising:

[0037] The first judgment unit is used to determine whether the driving lane of the autonomous vehicle meets the online calibration conditions;

[0038] The target detection unit is used to acquire road image data collected by the camera and laser point cloud data collected by the lidar of the autonomous vehicle when the online calibration conditions are met, and to perform target detection on the road image data and laser point cloud data to obtain image detection target and point cloud detection target.

[0039] The second judgment unit is used to determine whether the image detection target and the point cloud detection target are the same target;

[0040] The calibration calculation unit is used to acquire the first feature point data of the image detection target and the second feature point data of the point cloud detection target when they are the same target, and to perform matching calculations on the first feature point data and the second feature point data to obtain the online calibration result between the camera and the lidar.

[0041] Thirdly, embodiments of this application also provide an electronic device, including:

[0042] Processor; and

[0043] A memory is configured to store computer-executable instructions, which, when executed, cause the processor to perform any of the aforementioned sensor online calibration methods for autonomous vehicles.

[0044] Fourthly, embodiments of this application also provide a computer-readable storage medium storing one or more programs that, when executed by an electronic device including multiple applications, cause the electronic device to perform any of the aforementioned sensor online calibration methods for autonomous vehicles.

[0045] The above-mentioned at least one technical solution adopted in the embodiments of this application can achieve the following beneficial effects: When it is necessary to perform online calibration of the sensors of autonomous vehicles, the embodiments of this application first determine whether the driving lane of the autonomous vehicle meets the online calibration conditions. When the online calibration conditions are met, road image data and corresponding laser point cloud data are acquired and target detection is performed respectively. The target consistency judgment is performed on the detected image detection target and the point cloud detection target. When the two detected targets correspond to the same target, the relative extrinsic parameters of the camera and the lidar are calculated based on the feature points of the two detected targets to obtain the online calibration result.

[0046] This application embodiment uses the driving lane of the autonomous vehicle as a constraint condition. When this constraint condition is met, this application embodiment can accurately determine whether the image detection target and the point cloud detection target correspond to the same target. When it is determined that the image detection target and the point cloud detection target correspond to the same target, feature point matching can be performed accurately and quickly without complex feature point matching algorithms during the relative extrinsic parameter calculation process. Moreover, during the target detection process, the existing target detection algorithms for autonomous vehicles can be used, which can not only further improve the computational efficiency of online calibration, but also save computational resources. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0048] Figure 1 This is a flowchart illustrating an online sensor calibration method for an autonomous vehicle according to an embodiment of this application.

[0049] Figure 2 This is a schematic diagram of the lane boundary line detection results shown in the embodiments of this application;

[0050] Figure 3 This is a schematic diagram of road image data and image detection targets shown in the embodiments of this application;

[0051] Figure 4 This is a schematic diagram of laser point cloud data and its point cloud detection target shown in the embodiments of this application;

[0052] Figure 5 This is a schematic diagram of the structure of an online sensor calibration device for an autonomous vehicle according to an embodiment of this application;

[0053] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0056] This application provides an online sensor calibration method for autonomous vehicles, such as... Figure 1 The diagram shows a flowchart of an online sensor calibration method for an autonomous vehicle according to an embodiment of this application. The method includes at least the following steps S110 to S140:

[0057] Step S110: Determine whether the driving lane of the autonomous vehicle meets the online calibration conditions.

[0058] Step S120: When the online calibration conditions are met, acquire road image data collected by the camera of the autonomous vehicle and laser point cloud data collected by the lidar, and perform target detection on the road image data and laser point cloud data to obtain image detection target and point cloud detection target.

[0059] The online sensor calibration method for autonomous vehicles in this embodiment is executed by the autonomous vehicle itself. The vehicle is pre-installed with cameras and LiDAR, and after initial calibration, it can collect data in real time. When the initial extrinsic parameters of sensors such as cameras and LiDAR change and online calibration is needed to update or correct the relative extrinsic parameters between sensors, it is necessary to first determine whether the current driving lane meets the online calibration conditions. Only when the online calibration conditions are met will the road image data collected by the camera and the laser point cloud data collected by the LiDAR be acquired, ensuring that subsequent steps can be executed normally.

[0060] In addition, since the data acquisition frequencies of cameras and lidar are different, in order to ensure the accuracy of subsequent data processing, it is also necessary to perform time synchronization processing on road image data and lidar point cloud data, for example, by data acquisition through hardware triggering.

[0061] After collecting road image data and laser point cloud data, this embodiment can perform target detection on the road image data and laser point cloud data based on the existing target detection algorithm of autonomous vehicles, without the need to design an additional target detection algorithm, thus saving computing resources.

[0062] Step S130: Determine whether the image detection target and the point cloud detection target are the same target.

[0063] Online calibration needs to address two key issues: target matching and relative extrinsic parameter calculation between targets. If the first issue is resolved, the second issue is easily solved.

[0064] Therefore, after obtaining the detection targets of the two types of data, this application first determines whether the two detection targets correspond to the same target. When it is determined that the two detection targets correspond to the same target, the relative extrinsic parameters of the feature points of the two detection targets are calculated based on the feature point matching algorithm to obtain the online calibration result.

[0065] Step S140: When the targets are the same, acquire the first feature point data of the image detection target and the second feature point data of the point cloud detection target, perform matching calculation on the first feature point data and the second feature point data, and obtain the online calibration result between the camera and the lidar.

[0066] like Figure 1 As shown in the online calibration method, when the sensors of an autonomous vehicle need to be calibrated online, this embodiment first determines whether the driving lane of the autonomous vehicle meets the online calibration conditions. When the online calibration conditions are met, road image data and corresponding laser point cloud data are acquired and target detection is performed separately. The consistency of the detected image detection target and the point cloud detection target is judged. Only when the two detected targets are the same target, the relative extrinsic parameters of the camera and the LiDAR are calculated based on the feature points of the two detected targets to obtain the online calibration result.

[0067] This embodiment uses the driving lane of the autonomous vehicle as a constraint. When this constraint is met, this embodiment can accurately determine whether the image detection target and the point cloud detection target correspond to the same target. When it is determined that the image detection target and the point cloud detection target correspond to the same target, feature point matching can be performed accurately and quickly without complex feature point matching algorithms during the relative extrinsic parameter calculation process. Furthermore, during the target detection process, existing target detection algorithms for autonomous vehicles can be utilized, which can not only further improve the computational efficiency of online calibration but also save computational resources.

[0068] In some embodiments of this application, determining whether the driving lane of the autonomous vehicle meets the online calibration conditions includes:

[0069] Obtain the lane type of the current lane being driven by the autonomous vehicle;

[0070] If the autonomous vehicle is currently traveling in a straight lane, then the online calibration conditions are met; otherwise, the online calibration conditions are not met.

[0071] Considering that when an autonomous vehicle is driving in a straight lane, its cameras and LiDAR can collect complete data on targets directly in front of or behind the vehicle without occlusion issues, ensuring the accuracy of target detection and providing a data foundation for precise feature point matching; and that in a straight lane, using the autonomous vehicle as a reference, it is possible to accurately determine whether the image-detected target and the point cloud-detected target correspond to the target directly in front of or behind the vehicle, making target matching between different sensors simple and easy.

[0072] Therefore, this application specifically addresses the determination of whether an autonomous vehicle is traveling in a straight lane, such as... Figure 3 As shown, when an autonomous vehicle is traveling on a straight road and maintaining its position on the center line of the lane, it can determine that the target in front of or behind the vehicle captured by the vehicle's camera and LiDAR is the same target. Here, the target in front of or behind the vehicle mainly refers to the target directly in front of or behind the vehicle traveling in its lane. The detected targets include traffic participants such as vehicles. Therefore, this application uses the autonomous vehicle's driving lane as a constraint condition for online calibration. When this constraint condition is met, it can be determined that two subsequently detected targets correspond to the same target, solving the target matching problem mentioned above.

[0073] In some embodiments of this application, determining whether the image detection target and the point cloud detection target are the same target includes:

[0074] Determine whether the image detection target and the point cloud detection target are within the lane currently being traveled by the autonomous vehicle;

[0075] If the target is in the lane where the autonomous vehicle is currently traveling, then determine whether the image detection target and the point cloud detection target are both targets in front of the vehicle or both targets behind the vehicle.

[0076] If both are targets in front of the vehicle or both are targets behind the vehicle, the image detection target and the point cloud detection target are determined to be the same target.

[0077] Specifically, the determination of whether the image detection target is within the lane currently being traveled by the autonomous vehicle is achieved through the following steps:

[0078] Lane line detection is performed on the road image data to obtain the lane boundary line of the current driving lane of the autonomous vehicle. Here, the lane line detection algorithm can adopt the existing detection algorithm of autonomous vehicles to save computing resources.

[0079] The positional relationship between the image detection target and the lane boundary line is obtained. If the image detection target is within the lane boundary line, it is determined that the image detection target is within the lane currently being traveled by the autonomous vehicle.

[0080] like Figure 3 As shown, when an autonomous vehicle is driving on a straight road, it uses existing image target detection algorithms to detect targets in the road image data, resulting in the following: Figure 2 The 3D image target detection results shown, and the lane detection results obtained by using existing lane detection algorithms for autonomous vehicles on road image data, are as follows. Figure 3The double-sided boundary lines shown indicate that, based on location information, it is possible to determine whether the target detected in the 3D image is on the autonomous vehicle's lane.

[0081] In this embodiment, when obtaining the positional relationship between the image detection target and the lane boundary line, the image detection target and the lane boundary line should be transformed to the same coordinate system. For example, the pixels of the road image data are transformed to the vehicle coordinate system. Assuming (u, v) are the coordinates of the point projected onto the image plane, (c... x c y (f) is the reference point. x ,f y () is the focal length in pixels, X c Y c Z c It is the coordinate point in the camera coordinate system, X w Y w Z w These are coordinate points in the vehicle coordinate system, where the reference point can be the center point of the image.

[0082] According to the pinhole imaging formula:

[0083]

[0084] Derivation of formula (2):

[0085]

[0086]

[0087] So let's assume the height of an object on the ground is zero, i.e., Z. w =0, then the initial extrinsic parameter calibration matrix of the camera and lidar is used. Based on the following formula (3):

[0088]

[0089] The following formula (4) can be calculated:

[0090]

[0091]

[0092]

[0093] Formula (4) can be used to transform the lane line detection results in the image to the current vehicle coordinate system, and at the same time, the center pixel of the 3D image detection target can be projected to the front vehicle coordinate system. Then, based on the positional relationship, it can be determined whether the 3D image detection target is on the lane currently being driven by the autonomous vehicle.

[0094] In some embodiments of this application, the determination of whether the point cloud detection target is within the lane currently being traveled by the autonomous vehicle can be achieved through the following steps:

[0095] The lane boundary line of the lane currently being driven by the autonomous vehicle can be virtually generated based on the vehicle width information of the autonomous vehicle in the laser point cloud data; alternatively, high-precision map data can also be obtained to obtain the lane boundary line of the lane being driven by the autonomous vehicle based on the location information of the autonomous vehicle.

[0096] The positional relationship between the point cloud detection target and the virtual lane boundary line is obtained. If the point cloud detection target is within the virtual lane boundary line, it is determined that the point cloud detection target is within the lane currently being traveled by the autonomous vehicle.

[0097] Once it is determined that both the image detection target and the point cloud detection target are within the lane currently being traveled by the autonomous vehicle, the following steps can be used to determine whether the image detection target and the point cloud detection target are both targets in front of the vehicle or both targets behind the vehicle:

[0098] The positional relationship between the image detection target and the autonomous vehicle, and the positional relationship between the point cloud detection target and the autonomous vehicle are obtained.

[0099] If the image detection target and the point cloud detection target are located in front of the autonomous vehicle, then the image detection target and the point cloud detection target are both determined to be targets in front of the vehicle; if the image detection target and the point cloud detection target are located behind the autonomous vehicle, then the image detection target and the point cloud detection target are both determined to be targets behind the vehicle.

[0100] As illustrated in the embodiments of equations (1) to (4) above, after transforming the image detection target to the vehicle coordinate system, the positional relationship between the image detection target and the autonomous vehicle can be obtained, thereby determining whether the image detection target is a target in front of the vehicle or a target behind the vehicle. Since the point cloud detection targets detected by the existing point cloud detection algorithms of autonomous vehicles generally provide positions based on the vehicle coordinate system, therefore, as Figure 4 As shown, it is easy to determine which point cloud detection targets belong to the front of the vehicle and which point cloud detection targets belong to the rear of the vehicle.

[0101] In some embodiments of this application, the method further includes:

[0102] If there are multiple image detection targets and point cloud detection targets that are both in front of the vehicle and are located in the lane where the autonomous vehicle is currently traveling, then the image detection target and point cloud detection target that is closest to the autonomous vehicle is determined as the optimal target;

[0103] The process of acquiring the first feature point data of the image detection target and the second feature point data of the point cloud detection target includes:

[0104] Obtain the optimal first feature point data of the image detection target and the optimal second feature point data of the point cloud detection target.

[0105] In other words, when there are multiple image detection targets and point cloud detection targets in front of the vehicle, this embodiment preferably uses the target closest to the autonomous vehicle as the optimal target for relative extrinsic parameter calculation. This is because the target closest to the vehicle has the most complete features when it is collected by the sensor. Calculating relative extrinsic parameters based on the feature points of the optimal target can improve the accuracy of the calculation results.

[0106] Based on the above embodiments, it is possible to associate the image detection target and the point cloud detection target to achieve the matching of the detected target between the camera and the LiDAR. Then, the relative extrinsic parameters between the camera and the LiDAR can be calculated using the feature point matching algorithm in the prior art.

[0107] refer to Figure 3 and Figure 4 As is known, image target detection includes an image target detection box, and point cloud target detection includes a point cloud target detection box. Therefore, obtaining the first feature point data of the image target and the second feature point data of the point cloud target includes:

[0108] The corner data of the target detection box in the image are obtained as the first feature point data, and the corner data of the target detection box in the point cloud are obtained as the second feature point data. That is, the eight corner points of the target detection box in the image are used as the first feature point data, and the eight corner points of the target detection box in the point cloud are used as the second feature point data. The multi-point perspective imaging (PNP) algorithm is used to obtain the relative extrinsic parameters between the camera and the lidar based on the first feature point data and the second feature point data. The online calibration result is obtained based on the relative extrinsic parameters between the camera and the lidar.

[0109] Assume the coordinates of a 3D point in space are [X w Y w Z w ] T The homogeneous coordinates of this 3D point are represented as [X... w Y w Z w 1] T The coordinates of the projection point are [uv]. T The homogeneous coordinate system is [uv 1]. TIf the camera's intrinsic parameter matrix is ​​K, then the process of solving for the relative extrinsic parameters between the camera and the lidar, namely the rotation matrix R and the displacement vector t, based on the PNP algorithm is as follows:

[0110] The perspective projection model is as follows:

[0111]

[0112] Expanding equation (5), we get the following equation (6):

[0113]

[0114] The system of equations for formula (6) is as follows:

[0115] z c u c =f 11 X w +f 12 Y w +f 13 Z w +f 14

[0116] z c v c =f 21 X w +f 22 Y w +f 23 Z w +f 24

[0117] z c =f 31 X w +f 32 Y w +f 33 Z w +f 34 (7)

[0118] Eliminate equation (7) Z c After calculation, the following formula (8) is obtained:

[0119] f 11 X w +f 12 Y w +f 13 Z w +f 14 -f 31 X w u c +f 32 Y w u c +f 33Z w u c +f 34 u c =0

[0120] f 21 X w +f 22 Y w +f 23 Z w +f 24 -f 31 X w v c +f 32 Y w v c +f 33 Z w v c +f 34 v c =0 (8)

[0121] Each set of 3D-2D matching points corresponds to two equations, resulting in a total of 12 unknowns. At least 6 sets of matching points are required. If there are N sets of matching points, then...

[0122]

[0123] Rewriting the above equation (9) in matrix form, we get AF = 0. When N = 6, the linear equation system can be solved directly; when N ≥ 6, the least squares solution can be obtained, and singular value decomposition (SVD) can be performed on F, i.e., F = UDV. T .

[0124] Since F = [KR Kt], the estimated values ​​of the rotation matrix R and the translation vector t are as follows:

[0125]

[0126]

[0127] Based on the PNP algorithm shown in the above embodiments, it can be seen that after the matching of the image detection target and the point cloud detection target is completed, the relatively simple PNP algorithm can be used to complete the relative extrinsic parameter calculation. During the relative extrinsic parameter calculation process, it is not necessary to set initial values ​​to quickly calculate the rotation matrix and displacement vector, thereby achieving online calibration.

[0128] This application also provides an online sensor calibration device 500 for autonomous vehicles, such as... Figure 5The diagram shows a schematic of an online sensor calibration device for an autonomous vehicle according to an embodiment of this application. The device 500 includes: a first judgment unit 510, a target detection unit 520, a second judgment unit 530, and a calibration calculation unit 540, wherein:

[0129] The first judgment unit 510 is used to determine whether the driving lane of the autonomous vehicle meets the online calibration conditions;

[0130] The target detection unit 520 is used to acquire road image data collected by the camera and laser point cloud data collected by the lidar of the autonomous vehicle when the online calibration conditions are met, and to perform target detection on the road image data and laser point cloud data to obtain image detection target and point cloud detection target.

[0131] The second judgment unit 530 is used to determine whether the image detection target and the point cloud detection target are the same target;

[0132] The calibration calculation unit 540 is used to acquire the first feature point data of the image detection target and the second feature point data of the point cloud detection target when they are the same target, and to perform matching calculation on the first feature point data and the second feature point data to obtain the online calibration result between the camera and the lidar.

[0133] In one embodiment of this application, the first determination unit 510 is specifically used to obtain the lane type of the lane in which the autonomous vehicle is currently driving; if the autonomous vehicle is currently driving in a straight lane, it is determined that the online calibration conditions are met, otherwise the online calibration conditions are not met.

[0134] In one embodiment of this application, the second determination unit 530 is specifically used to determine whether the image detection target and the point cloud detection target are within the lane currently being traveled by the autonomous vehicle; if they are within the lane currently being traveled by the autonomous vehicle, then it is determined whether the image detection target and the point cloud detection target are both targets in front of the vehicle or both targets behind the vehicle; if they are both targets in front of the vehicle or both targets behind the vehicle, then it is determined that the image detection target and the point cloud detection target are the same target.

[0135] In one embodiment of this application, the second determination unit 530 includes a lane determination module and a vehicle front and rear determination module;

[0136] The lane determination module is used to perform lane line detection on the road image data to obtain the lane boundary line of the lane currently being traveled by the autonomous vehicle; obtain the positional relationship between the image detection target and the lane boundary line, and if the image detection target is within the lane boundary line, then it is determined that the image detection target is within the lane currently being traveled by the autonomous vehicle; and, based on the vehicle width information of the autonomous vehicle in the laser point cloud data, virtually construct the lane boundary line of the lane currently being traveled by the autonomous vehicle; obtain the positional relationship between the point cloud detection target and the virtual lane boundary line, and if the point cloud detection target is within the virtual lane boundary line, then it is determined that the point cloud detection target is within the lane currently being traveled by the autonomous vehicle.

[0137] The vehicle-front / vehicle-rear judgment module is used to obtain the positional relationship between the image detection target and the autonomous vehicle, and the positional relationship between the point cloud detection target and the autonomous vehicle; if the image detection target and the point cloud detection target are located in front of the autonomous vehicle, then the image detection target and the point cloud detection target are determined to be both targets in front of the vehicle; if the image detection target and the point cloud detection target are located behind the autonomous vehicle, then the image detection target and the point cloud detection target are determined to be both targets behind the vehicle.

[0138] In one embodiment of this application, the second determination unit 530 is further configured to determine the image detection target and point cloud detection target that are both targets in front of the vehicle and are located in the lane where the autonomous vehicle is currently traveling as the optimal target if there are multiple such targets.

[0139] The calibration calculation unit 540 is also used to obtain the optimal first feature point data of the image detection target and the optimal second feature point data of the point cloud detection target.

[0140] In one embodiment of this application, the image detection target includes an image target detection box, and the point cloud detection target includes a point cloud target detection box. The calibration calculation unit 540 is specifically used to acquire corner data of the image target detection box as the first feature point data and acquire corner data of the point cloud target detection box as the second feature point data. The multi-point perspective imaging (PNP) algorithm is used to obtain the relative extrinsic parameters between the camera and the lidar based on the first feature point data and the second feature point data. The online calibration result is obtained based on the relative extrinsic parameters between the camera and the lidar.

[0141] It is understood that the above-mentioned online sensor calibration device for autonomous vehicles can implement each step of the online sensor calibration method for autonomous vehicles provided in the foregoing embodiments. The relevant explanations of the online sensor calibration method for autonomous vehicles are applicable to the online sensor calibration device for autonomous vehicles, and will not be repeated here.

[0142] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 6 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0143] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0144] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0145] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a high-precision map update mechanism at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0146] Determine whether the driving lane of the autonomous vehicle meets the online calibration conditions;

[0147] When the online calibration conditions are met, road image data collected by the camera of the autonomous vehicle and laser point cloud data collected by the lidar are acquired. Target detection is performed on the road image data and laser point cloud data to obtain image detection targets and point cloud detection targets.

[0148] Determine whether the image detection target and the point cloud detection target are the same target;

[0149] When the targets are the same, the first feature point data of the image detection target and the second feature point data of the point cloud detection target are obtained. The first feature point data and the second feature point data are matched and calculated to obtain the online calibration result between the camera and the lidar.

[0150] The above is as stated in this application. Figure 1 The method executed by the sensor online calibration device for autonomous vehicles disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0151] The electronic device can also perform Figure 1 The method for implementing the online sensor calibration device of an autonomous vehicle is described, and the implementation of the online sensor calibration device of an autonomous vehicle is described. Figure 1 The functions of the embodiments shown are not described again in this application.

[0152] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 1 The method executed by the sensor online calibration device of the autonomous vehicle in the illustrated embodiment is specifically used to perform:

[0153] Determine whether the driving lane of the autonomous vehicle meets the online calibration conditions;

[0154] When the online calibration conditions are met, road image data collected by the camera of the autonomous vehicle and laser point cloud data collected by the lidar are acquired. Target detection is performed on the road image data and laser point cloud data to obtain image detection targets and point cloud detection targets.

[0155] Determine whether the image detection target and the point cloud detection target are the same target;

[0156] When the targets are the same, the first feature point data of the image detection target and the second feature point data of the point cloud detection target are obtained. The first feature point data and the second feature point data are matched and calculated to obtain the online calibration result between the camera and the lidar.

[0157] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0158] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0159] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0160] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0161] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0162] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0163] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0164] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0165] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0166] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A sensor online calibration method of an autonomous vehicle, characterized in that, The method includes: Determine whether the driving lane of the autonomous vehicle meets the online calibration conditions; When the online calibration conditions are met, road image data collected by the camera of the autonomous vehicle and laser point cloud data collected by the lidar are acquired. Target detection is performed on the road image data and laser point cloud data to obtain image detection targets and point cloud detection targets. Determine whether the image detection target and the point cloud detection target are the same target; When the target is the same, the first feature point data of the image detection target and the second feature point data of the point cloud detection target are obtained. The first feature point data and the second feature point data are matched and calculated to obtain the online calibration result between the camera and the lidar. Determining whether the image detection target and the point cloud detection target are the same target includes: Determine whether the image detection target and the point cloud detection target are within the lane currently being traveled by the autonomous vehicle; If the target is in the lane where the autonomous vehicle is currently traveling, then determine whether the image detection target and the point cloud detection target are both targets in front of the vehicle or both targets behind the vehicle. If both are targets in front of the vehicle or both are targets behind the vehicle, then the image-detected target and the point cloud-detected target are determined to be the same target; Determining whether the image detection target and the point cloud detection target are within the lane currently being traveled by the autonomous vehicle includes: Lane line detection is performed on the road image data to obtain the lane boundary line of the lane currently being traveled by the autonomous vehicle; The positional relationship between the image detection target and the lane boundary line is obtained. If the image detection target is within the lane boundary line, it is determined that the image detection target is within the lane currently being traveled by the autonomous vehicle. In addition, the lane boundary line of the current driving lane of the autonomous vehicle is virtually generated based on the vehicle width information of the autonomous vehicle in the laser point cloud data. The positional relationship between the point cloud detection target and the virtual lane boundary line is obtained. If the point cloud detection target is within the virtual lane boundary line, it is determined that the point cloud detection target is within the lane currently being traveled by the autonomous vehicle.

2. The method of claim 1, wherein, Determining whether the driving lane of the autonomous vehicle meets the online calibration conditions includes: Obtain the lane type of the current lane being driven by the autonomous vehicle; If the autonomous vehicle is currently traveling in a straight lane, then the online calibration conditions are met; otherwise, the online calibration conditions are not met.

3. The method of claim 1, wherein, Determining whether the image detection target and the point cloud detection target are both in front of the vehicle or both behind the vehicle includes: The positional relationship between the image detection target and the autonomous vehicle, and the positional relationship between the point cloud detection target and the autonomous vehicle are obtained. If the image detection target and the point cloud detection target are located in front of the autonomous vehicle, then the image detection target and the point cloud detection target are both determined to be targets in front of the vehicle. If the image detection target and the point cloud detection target are located behind the autonomous vehicle, then the image detection target and the point cloud detection target are both determined to be targets behind the vehicle.

4. The method of claim 1, wherein, The method further includes: If there are multiple image detection targets and point cloud detection targets that are both in front of the vehicle and are located in the lane where the autonomous vehicle is currently traveling, then the image detection target and point cloud detection target that is closest to the autonomous vehicle is determined as the optimal target; The process of acquiring the first feature point data of the image detection target and the second feature point data of the point cloud detection target includes: Obtain the optimal first feature point data of the image detection target and the optimal second feature point data of the point cloud detection target.

5. The method of claim 1, wherein, The image detection target includes an image target detection box, and the point cloud detection target includes a point cloud target detection box. The process of acquiring first feature point data of the image detection target and second feature point data of the point cloud detection target, and performing matching calculations on the first and second feature point data to obtain the online calibration result between the camera and the lidar includes: The corner data of the target detection box in the image is obtained as the first feature point data, and the corner data of the target detection box in the point cloud is obtained as the second feature point data. The relative extrinsic parameters between the camera and the lidar are obtained by using the multi-point perspective imaging (PNP) algorithm on the first feature point data and the second feature point data. The online calibration result is obtained based on the relative extrinsic parameters between the camera and the lidar.

6. A sensor online calibration device of an autonomous vehicle, characterized in that, The device includes: The first judgment unit is used to determine whether the driving lane of the autonomous vehicle meets the online calibration conditions; The target detection unit is used to acquire road image data collected by the camera and laser point cloud data collected by the lidar of the autonomous vehicle when the online calibration conditions are met, and to perform target detection on the road image data and laser point cloud data to obtain image detection target and point cloud detection target. The second judgment unit is used to determine whether the image detection target and the point cloud detection target are the same target; The calibration calculation unit is used to acquire the first feature point data of the image detection target and the second feature point data of the point cloud detection target when they are the same target, and to perform matching calculations on the first feature point data and the second feature point data to obtain the online calibration result between the camera and the lidar. The second judgment unit is specifically used for: Determine whether the image detection target and the point cloud detection target are within the lane currently being traveled by the autonomous vehicle; If the target is in the lane where the autonomous vehicle is currently traveling, then determine whether the image detection target and the point cloud detection target are both targets in front of the vehicle or both targets behind the vehicle. If both are targets in front of the vehicle or both are targets behind the vehicle, then the image-detected target and the point cloud-detected target are determined to be the same target; The second judgment unit is specifically used for: Lane line detection is performed on the road image data to obtain the lane boundary line of the lane currently being traveled by the autonomous vehicle; The positional relationship between the image detection target and the lane boundary line is obtained. If the image detection target is within the lane boundary line, it is determined that the image detection target is within the lane currently being traveled by the autonomous vehicle. In addition, the lane boundary line of the current driving lane of the autonomous vehicle is virtually generated based on the vehicle width information of the autonomous vehicle in the laser point cloud data. The positional relationship between the point cloud detection target and the virtual lane boundary line is obtained. If the point cloud detection target is within the virtual lane boundary line, it is determined that the point cloud detection target is within the lane currently being traveled by the autonomous vehicle.

7. An electronic device, comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method of any one of claims 1 to 5.

8. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the method of any one of claims 1 to 5.

Images