Multi-sensor joint calibration method and device and computer readable storage medium

Abstract

The invention provides a multi-sensor joint calibration method and device and a computer readable storage medium, and the method comprises the steps: obtaining a first target frame obtained after a target sensor senses a target object, and a plurality of second target frames obtained after a plurality of to-be-calibrated sensors sense the target object, and determining an initialized extrinsic parameter homogeneous matrix between each sensor to be calibrated and the target sensor according to a matching degree mean value maximization strategy of the first target frame and the second target frame. The coordinate system of the target sensor is set as a fixed reference system by taking the target sensor as a reference, and the data sensed by the to-be-calibrated sensor is converted into the coordinate system of the target sensor for calibration, so that the complexity caused by intermediate calibration transmission is reduced, error accumulation caused by cascade calibration is avoided, and the calibration accuracy is improved.

Description

Technical Field

[0001] The present application relates to the technical field of sensor calibration, and in particular to a multi-sensor joint calibration method, device and computer-readable storage medium. Background Art

[0002] Autonomous driving technology is gaining more and more attention in industry and academia. Safe and reliable driving depends on accurate sensor perception. However, in the real world, due to the complex driving environment, the perception results of a single sensor cannot fully guarantee the integrity and reliability of environmental detection. Therefore, in order to ensure that autonomous vehicles can operate in various changing environments, multi-model complementary sensor collaboration technology is applied in this field. This technology can provide sufficient environmental information to achieve more reliable data fusion, thereby improving the environmental perception and safety of autonomous vehicles.

[0003] To realize the multi-sensor perception and positioning system of intelligent vehicles, accurate sensor calibration is essential. Sensor internal parameter calibration can obtain the mapping relationship inside the sensor, while external parameter calibration can convert multiple sensors to the same spatial coordinate system. Generally speaking, since the parameters such as yaw angle, pitch angle and roll angle of the sensor may change compared with the digital model after installation, most sensors need to be calibrated to ensure the measurement accuracy of the sensor.

[0004] At present, when performing external parameter calibration of multiple sensors, cascade calibration is performed using sensors with overlapping field of view angles. This will lead to the accumulation of calibration errors between different sensors and increase the final calibration error, thereby affecting the results and accuracy of sensor perception. In addition, this method has certain requirements on the field of view overlap between sensors. If the field of view overlap is small, it will lead to increased calibration difficulty and error, and ultimately affect the calibration accuracy. Summary of the invention

[0005] To overcome the problems existing in the related art, this specification provides a multi-sensor joint calibration method, device and computer-readable storage medium.

[0006] According to a first aspect of an embodiment of this specification, a multi-sensor joint calibration method is provided, the method comprising:

[0007] Acquire a first target frame obtained after the target sensor perceives the target object and a plurality of second target frames obtained after a plurality of sensors to be calibrated perceive the target object respectively;

[0008] Determining an initialized extrinsic parameter homogeneous matrix between each of the sensors to be calibrated and the target sensor according to a strategy for maximizing the mean value of the matching degree between the first target frame and the second target frame;

[0009] In a calibration coordinate system established with the target sensor as a reference, a joint calibration extrinsic parameter homogeneous matrix between the multiple sensors to be calibrated is established based on each of the initialized extrinsic parameter homogeneous matrices.

[0010] According to a multi-sensor joint calibration method provided by the present application, in a calibration coordinate system established based on the target sensor as a reference, the calibration coordinate system includes point cloud data sensed by the target sensor and point cloud data of the sensor to be calibrated that has completed calibration.

[0011] According to a multi-sensor joint calibration method provided by the present application, when the target sensor is a target laser radar and the sensor to be calibrated is a laser radar to be calibrated,

[0012] The step of determining an initialized extrinsic parameter homogeneous matrix between the sensor to be calibrated and the target sensor according to a strategy of maximizing the mean matching degree between the first target frame and the second target frame includes:

[0013] Transforming the second target frame into the coordinate system of the target sensor through an initial transformation matrix;

[0014] Calculate the average matching degree of all converted second target frames with the first target frame in the overlapping field of view of the target laser radar and the laser radar to be calibrated, and obtain a matching score;

[0015] When the matching score meets the termination condition, the initial transformation matrix is ​​output as an initialized external parameter homogeneous matrix between the laser radar to be calibrated and the target laser radar.

[0016] According to a multi-sensor joint calibration method provided by the present application, when the matching score does not meet the termination condition, the initial transformation matrix is ​​adjusted, and the step of determining the initialized extrinsic parameter homogeneous matrix between the sensor to be calibrated and the target sensor using the strategy of maximizing the mean matching degree of the first target frame and the second target frame is repeated until the termination condition is met.

[0017] According to a multi-sensor joint calibration method provided by the present application, when the target sensor is a target laser radar and the sensor to be calibrated is a camera to be calibrated,

[0018] The step of determining an initialized extrinsic parameter homogeneous matrix between the sensor to be calibrated and the target sensor according to a strategy of maximizing the mean matching degree between the first target frame and the second target frame includes:

[0019] Mapping the first target frame to the coordinate system of the camera to be calibrated through an intrinsic parameter matrix and an extrinsic parameter matrix; the intrinsic parameter matrix is ​​determined according to the properties of the camera to be calibrated;

[0020] Matching and comparing the mapped target frame with the second target frame;

[0021] When the matching result meets the termination condition, the external parameter matrix is ​​output as the initialized external parameter homogeneous matrix between the camera to be calibrated and the target laser radar.

[0022] According to a multi-sensor joint calibration method provided by the present application, before acquiring a first target frame obtained after the target sensor perceives the target object and a plurality of second target frames obtained after the plurality of sensors to be calibrated perceive the target object respectively, the method further includes:

[0023] The first point cloud data obtained after the target sensor senses the target object and the second point cloud data obtained after the sensor to be calibrated senses the target object are obtained.

[0024] According to a multi-sensor joint calibration method provided by the present application, the method further includes:

[0025] Mapping the first point cloud data to the coordinate system of the camera to be calibrated through an intrinsic parameter matrix and an extrinsic parameter matrix;

[0026] Analyzing the matching between the outline of the mapped first point cloud data and the outline of the second target frame;

[0027] When the matching result meets the termination condition, the external parameter matrix is ​​output as the initialized external parameter homogeneous matrix between the camera to be calibrated and the target laser radar.

[0028] According to a multi-sensor joint calibration method provided by the present application, when the target sensor is a target laser radar and the sensor to be calibrated is a millimeter-wave radar to be calibrated,

[0029] The method further comprises:

[0030] Feature matching is performed on the first point cloud data and the second point cloud data to determine an initialized extrinsic parameter homogeneous matrix between the target laser radar and the millimeter-wave radar to be calibrated.

[0031] The present application also provides a multi-sensor joint calibration device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the multi-sensor joint calibration methods described above is implemented.

[0032] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the multi-sensor joint calibration method as described in any one of the above is implemented.

[0033] In the embodiment of this specification, the multi-sensor joint calibration method, device and computer-readable storage medium are compared with the current cascade calibration using sensors with overlapping field of view angles, which leads to the accumulation of calibration errors between different sensors and affects the results and accuracy of sensor perception. The first target frame obtained after the target sensor perceives the target object and the multiple second target frames obtained after the multiple sensors to be calibrated perceive the target object respectively, and the initialization external parameter homogeneous matrix between the sensor to be calibrated and the target sensor is determined according to the strategy of maximizing the mean value of the matching degree between the first target frame and the second target frame. By setting the coordinate system of the target sensor as a fixed reference system based on the target sensor, the data perceived by the sensor to be calibrated is converted to the coordinate system of the target sensor for calibration, which reduces the complexity brought by the intermediate calibration transfer, avoids the error accumulation caused by the cascade calibration, and provides the accuracy of the calibration. Then, in the calibration coordinate system established based on the target sensor, based on each of the initialization external parameter homogeneous matrices, a joint calibration external parameter homogeneous matrix between the multiple sensors to be calibrated is established to complete the calibration of multiple sensors to be calibrated of the entire perception system, thereby improving the accuracy of the final calibration result.

[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0036] Figure 1 is a flowchart of a multi-sensor joint calibration method according to an exemplary embodiment of the present specification;

[0037] Figure 2 is a schematic diagram of multi-sensor installation on a real vehicle according to an exemplary embodiment of this specification;

[0038] Figure 3 It is a TF tree diagram of multi-sensor extrinsic parameter calibration of a real vehicle shown in this specification according to an exemplary embodiment;

[0039] Figure 4 is a calibration flow chart of a laser radar to be calibrated in a multi-sensor joint calibration method according to an exemplary embodiment of this specification;

[0040] Figure 5 is a schematic diagram of conversion between a camera coordinate system and an image coordinate system in a multi-sensor joint calibration method according to an exemplary embodiment of this specification;

[0041] Figure 6 is a schematic diagram of a multi-sensor joint calibration device according to an exemplary embodiment of the present specification;

[0042] Figure 7 It is a schematic block diagram of a multi-sensor joint calibration device according to an exemplary embodiment of this specification. DETAILED DESCRIPTION

[0043] Here, the technical solutions in the embodiments (or "implementations") of the present application will be described clearly and completely in conjunction with the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0044] If there are terms involving directional indications or positional relationships in the embodiments of the present application (such as up, down, left, right, front, back, inside, outside, top, bottom, center, vertical, horizontal, longitudinal, transverse, length, width, counterclockwise, clockwise, axial, radial, circumferential, etc.), such terms are only used to explain the relative positional relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings); if the specific posture changes, the directional indication or positional relationship will also change accordingly. In addition, the terms "first", "second", etc. involved in the embodiments of the present application are only used for the purpose of convenience of description and cannot be understood as indicating or implying relative importance.

[0045] The present application provides a multi-sensor joint calibration method, device and computer-readable storage medium. The present application is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0046] To realize the multi-sensor perception and positioning system of intelligent vehicles, accurate sensor calibration is essential. Sensor internal parameter calibration can obtain the mapping relationship inside the sensor, while external parameter calibration can convert multiple sensors to the same spatial coordinate system. Generally speaking, since the parameters such as yaw angle, pitch angle and roll angle of the sensor may change compared with the digital model after installation, most sensors need to be calibrated to ensure the measurement accuracy of the sensor.

[0047] Sensor calibration is an essential part of the autonomous driving perception system and a necessary step and prerequisite for subsequent sensor fusion. Its purpose is to transform multiple sensors into the same space-time coordinate system to make sensor fusion meaningful. It is a key prerequisite for decision-making and control.

[0048] Existing external parameter calibration technologies mostly use the same field of view between sensors and perform cascaded external parameter calibration between multiple sensors of a vehicle based on deep learning or collecting multi-frame point cloud data and matching point clouds with related graphs. This will lead to the accumulation of calibration errors between different sensors and increase the final calibration error, thereby affecting the results and accuracy of sensor perception. In addition, the cascaded external parameter calibration method has certain requirements for the field of view overlap between sensors. If the field of view overlap is small, it will lead to increased calibration difficulty and error, and ultimately affect the calibration accuracy.

[0049] In order to solve the above technical problems, this specification provides a multi-sensor joint calibration method.

[0050] By using the same target sensor as a reference for multiple sensors to be calibrated, a calibration coordinate system is established based on the target sensor, and finally the joint calibration extrinsic parameter homogeneous matrix between the sensors is obtained through TF coordinate conversion. After completing the multi-sensor extrinsic parameter calibration of the entire perception system, the target sensor can be removed for calibration of other vehicles.

[0051] This method directly uses the target LiDAR as a fixed reference system, reducing the complexity caused by intermediate transmission. It can reduce the cumulative error generated by the mutual calibration of multiple sensors in the traditional calibration method and improve the calibration accuracy. The target LiDAR can be reused, and there is no field of view overlap requirement for the sensor scheme of the perception system. It is suitable for all vehicles with calibration requirements, effectively reducing costs. The effectiveness and robustness of this method can be experimentally verified on autonomous driving vehicles with different sensor configurations.

[0052] The application scenario of the vehicle multi-sensor joint calibration described in this article is not only the above-mentioned vehicle sensor position offset, which requires joint calibration between multiple sensors, but also includes various vehicle multi-sensor joint calibration scenarios. No specific limitation is made here.

[0053] Among various sensors, especially cameras, LiDAR and millimeter-wave radar, they are widely used in the solution layout of autonomous vehicles. The following is an explanation of joint calibration using cameras, LiDAR and millimeter-wave radar as examples of sensors to be calibrated.

[0054] Figure 1 : is a flowchart of a multi-sensor joint calibration method provided in an embodiment of this specification, comprising the following steps:

[0055] S100, obtaining a first target frame obtained after a target sensor perceives a target object and a plurality of second target frames obtained after a plurality of sensors to be calibrated perceive the target object respectively.

[0056] The target sensor described in this article refers to the sensor used as a calibration reference in the multi-sensor, which can be any one of a camera sensor, a lidar sensor, and a millimeter wave sensor. The installation of the target sensor is only used for calibration, and it can be disassembled after the joint calibration of the multi-sensor is completed. Multiple sensors to be calibrated refer to two or more sensors that need to be calibrated together, which can be calibration between sensors of the same type (multiple sensors to be calibrated are all camera sensors) or calibration between sensors of different types (multiple sensors to be calibrated are camera sensors and lidar sensors). It should be noted that when selecting different sensors as target sensors, their own characteristics and limitations need to be considered, and corresponding measures should be taken to improve the accuracy and reliability of the calibration. For example, when a camera is used as a reference, the camera can provide rich texture and color information, which helps to identify and match features. However, the camera is easily affected by changes in light and weather, and its measurement accuracy is usually not as good as that of lidar and millimeter wave radar in the depth direction, so accurate internal parameter calibration and feature extraction algorithm optimization are required. For another example, when a millimeter wave radar is used as a reference, its advantage is that it can work in various weather conditions and is more accurate in speed measurement. However, the resolution of millimeter-wave radar is relatively low, which may affect the accuracy of calibration, so multi-frame data fusion is required to improve measurement accuracy. The final choice of which sensor to use as the target sensor depends on the specific application scenario, the performance characteristics of the sensor, and the requirements of the system.

[0057] In this paper, the target sensor needs to ensure the characteristics of the point cloud in the calibration scene, and then ensure the accuracy of the calibration external parameters. Therefore, according to the characteristics of the natural scene, a virtual address based on the high-beam medium- and long-range laser radar is constructed as a benchmark for calibration, which can not only reduce the cumulative error generated by the traditional calibration method when multiple sensors are calibrated with each other, but also reuse the target sensor. Other short-range laser radars with low beams but high accuracy, or radars with special properties (such as high resolution, strong anti-interference ability, etc.) may also be used for such calibration schemes under certain specific conditions. No specific restrictions are made here.

[0058] The following description takes the target sensor as a target laser radar as an example, and other embodiments are basically the same and will not be repeated here.

[0059] like Figure 2 As shown, Figure 2 This is a schematic diagram of the multi-sensor installation on the actual vehicle. A high-beam mid-range and long-range mechanical laser radar is installed on the top of the actual vehicle as the target laser radar, and it is ensured that the target laser radar point cloud can fully cover the vehicle's surroundings at 360° and have a long field of view.

[0060] Afterwards, the calibration scene is collected by real vehicle and multi-sensor joint calibration is performed, such as Figure 3 As shown, Figure 3 TF tree diagram for calibrating extrinsic parameters of multiple sensors on a real vehicle.

[0061] It is worth noting that the virtual address based on the high-beam mid-range and long-range LiDAR constructed by this multi-sensor joint calibration method can not only achieve the above-mentioned reuse and reduce the cumulative error generated by the traditional calibration method when multiple sensors are calibrated with each other, but also has no field of view overlap requirements for the sensors to be calibrated in the perception system. It is applicable to all vehicles with calibration requirements and effectively reduces costs. The effectiveness and robustness of this method can be experimentally verified on autonomous driving vehicles with different sensor configurations.

[0062] In this embodiment, the first target frame and the second target frame perceived by the target sensor and the sensor to be calibrated are used as matching evaluation indicators, which is more accurate for determining whether they are the same target after perception and fusion.

[0063] It can be understood that for different sensors to be calibrated, the corresponding target frames have slightly different shapes. For example, when calibrating between the target laser radar and the laser radar to be calibrated, the first target frame and the second target frame are 3D frames, and the matching degree is evaluated based on the perceived 3D frames. For another example, when calibrating between the target laser radar and the camera to be calibrated, since the laser radar has outstanding advantages such as high precision, long range, good real-time performance, and rich information collection, it can perceive the 3D entity of the target object, while the target object perceived by the pixel (image) coordinate system of the camera to be calibrated is planar. Therefore, when calibrating between the target laser radar and the camera to be calibrated, the first target frame perceived is a 3D frame, and the second target frame is a 2D frame. Two or more sensors are transformed into the same coordinate system, and then the transformed target frames are evaluated for matching degree.

[0064] S200 , determining an initialized extrinsic parameter homogeneous matrix between the sensor to be calibrated and the target sensor according to a strategy for maximizing the mean value of the matching degree between the first target frame and the second target frame.

[0065] The essence of sensor extrinsic calibration is to obtain the displacement (x, y, z) and rotation (roll, pitch, yaw) of the two sensors. The three-dimensional space can be described by a homogeneous transformation matrix T.

[0066] The first target frame and the second target frame are transformed to be in the same coordinate system. By adjusting the parameters of the initialized external parameter homogeneous matrix, the mean of the matching degree of all target frames is maximized. When the mean of the matching degree reaches the maximum / threshold value or meets certain convergence conditions, the external parameter homogeneous matrix at this time is the initialized external parameter homogeneous matrix for calibration between the sensor to be calibrated and the target sensor. This means that the data sensed by the sensor to be calibrated can be transformed into the target sensor coordinate system through the initialized external parameter homogeneous matrix, representing the same target object.

[0067] Optimization algorithms, such as gradient descent, can usually be used to adjust the parameters of the extrinsic homogeneous matrix.

[0068] In this embodiment, the calibration between the target sensor and the sensor to be calibrated is determined based on the matching degree of the first target frame and the second target frame perceived by the target sensor and the sensor to be calibrated. Since the post-target fusion algorithm is based on the target frame matching degree as the evaluation criterion for whether it is the same target, the external parameter homogeneous matrix obtained by the multi-sensor joint calibration method is more accurate for the lidar fusion algorithm to determine whether it is the same target.

[0069] S300 , in a calibration coordinate system established with the target sensor as a reference, establishing a joint calibration extrinsic parameter homogeneous matrix between the plurality of sensors to be calibrated based on the respective initialized extrinsic parameter homogeneous matrices.

[0070] Through the above process, the initialized extrinsic parameter homogeneous matrix between each sensor to be calibrated and the target sensor as the calibration reference is obtained, and each sensor to be calibrated is converted to the calibration coordinate system established with the target sensor as the reference based on the initialized extrinsic parameter homogeneous matrix obtained above. In the same coordinate system, a joint calibration extrinsic parameter homogeneous matrix between multiple sensors to be calibrated is established.

[0071] Specifically, if two sensors to be calibrated are jointly calibrated, the initialized extrinsic parameter homogeneous matrices corresponding to the two sensors to be calibrated are multiplied to obtain the joint calibration extrinsic parameter homogeneous matrix of the two sensors to be calibrated. Assume that the extrinsic parameter homogeneous matrix of the sensor to be calibrated A relative to the target laser radar is H AT , the homogeneous external parameter matrix of the sensor B to be calibrated relative to the target lidar is H BT , then the homogeneous matrix H of the external parameters of the sensor A to be calibrated relative to the sensor B to be calibrated can be calculated AB For: H AB =H BT -1 *H AT .

[0072] After completing the multi-sensor extrinsic calibration of the entire perception system, the target sensor can be removed for calibration of other vehicles.

[0073] In this embodiment, the overlap between each sensor to be calibrated and the target sensor is ensured by the above method, and one acquisition can be used for the calibration of multiple sensors, which greatly saves labor costs. In addition, the solution always takes the target sensor as the benchmark, avoids repeated calibration between multiple sensors such as multiple cameras and multiple radars, reduces the accumulation of errors, and improves the accuracy of calibration.

[0074] In some embodiments, in the calibration coordinate system established based on the target sensor as a reference, the calibration coordinate system includes the point cloud data sensed by the target sensor and the point cloud data of the sensor to be calibrated that has been calibrated. It can be understood that after calibrating the current sensor to be calibrated, a calibration coordinate system is established based on the target sensor, and the coordinate system of the target sensor is set as a fixed reference frame. The point cloud data of other sensors will be converted to the coordinate system of this target sensor to integrate more point cloud information to increase the number of point clouds of characteristic targets in the calibration scene. For example, the point clouds of other laser radars that have been calibrated can be transferred to the target laser radar coordinate system through coordinate conversion, and then merged with the point cloud of the target laser radar itself, thereby providing richer and more comprehensive point cloud data for calibration to ensure the characteristics of the target point cloud in the scene.

[0075] In other embodiments, the calibration method may be constructed as a calibration room, and the target sensor may be installed in the calibration room.

[0076] The calibration room described in this article is a spatial environment designed and constructed specifically for calibration operations. Installing the target sensor in the calibration room can better control environmental factors during the calibration process, such as light, temperature, humidity, etc., and reduce the interference of these factors on the measurement and calibration results. The calibration room provides a standardized space so that the conditions and settings of each calibration are as consistent as possible, which is conducive to improving the repeatability and reliability of the calibration results. The calibration room can be equipped with special positioning devices and markers to help more accurately determine the position and attitude of the target lidar, thereby providing a more accurate benchmark for the calibration of other radars. In summary, installing the target sensor in the calibration room helps to improve the accuracy, stability and repeatability of the entire multi-sensor joint calibration method.

[0077] The present application provides a multi-sensor joint calibration method, device and computer-readable storage medium. Compared with the current cascade calibration using sensors with overlapping field of view angles, which leads to the accumulation of calibration errors between different sensors and affects the results and accuracy of sensor perception, the first target frame obtained after the target sensor perceives the target object and the multiple second target frames obtained after the multiple sensors to be calibrated perceive the target object respectively, and the initialization external parameter homogeneous matrix between each of the sensors to be calibrated and the target sensor is determined according to the strategy of maximizing the mean value of the matching degree between the first target frame and the second target frame. By setting the coordinate system of the target sensor as a fixed reference system based on the target sensor, the data perceived by the sensor to be calibrated is converted to the coordinate system of the target sensor for calibration, which reduces the complexity brought by the intermediate calibration transfer, avoids the error accumulation caused by the cascade calibration, and improves the accuracy of calibration. Then, in the calibration coordinate system established with the target sensor as the reference, a joint calibration extrinsic parameter homogeneous matrix between the multiple sensors to be calibrated is established based on each of the initialized extrinsic parameter homogeneous matrices to complete the calibration of multiple sensors to be calibrated of the entire perception system, thereby improving the accuracy of the final calibration result.

[0078] Based on the above first embodiment, a second embodiment of a multi-sensor joint calibration method is proposed. The second embodiment is used to illustrate a calibration method when both the target sensor and the sensor to be calibrated are laser radars.

[0079] Specific, combined Figure 4 As shown in the figure, the calibration process for each laser radar to be calibrated is as follows:

[0080] Step a1, acquiring first point cloud data obtained after the target sensor perceives the target object and acquiring second point cloud data obtained after the sensor to be calibrated perceives the target object.

[0081] Step a2: identifying the target object in the acquired first point cloud data and the acquired second point cloud data respectively through a target detection algorithm.

[0082] Step a3, obtaining a first target frame obtained after the target sensor perceives the target object and a second target frame obtained after a plurality of sensors to be calibrated perceive the target object respectively.

[0083] Specifically, 3D bounding boxes are generated for these target objects based on the coordinates, size and orientation of each target object relative to the sensor, wherein the target laser radar generates a first target 3D box, and the laser radar to be calibrated generates a second target 3D box.

[0084] Step a4: determining an initialized extrinsic parameter homogeneous matrix between the sensor to be calibrated and the target sensor according to a strategy of maximizing the mean value of the matching degree between the first target frame and the second target frame.

[0085] Specifically, the process includes the following steps:

[0086] Step a41, convert the second target frame into the coordinate system of the target sensor through the initial transformation matrix. The initial transformation T is obtained according to the sensor installation digital model, which contains detailed information about the installation position and posture of the laser radar to be calibrated on the vehicle. Based on this, an initial transformation matrix T is preliminarily determined through mathematical calculation or preset model. This initial transformation matrix T preliminarily describes the position and direction relationship of the laser radar to be calibrated relative to the target laser radar, which needs to be optimized later.

[0087] In this step, the initial transformation matrix T is used to transform the coordinates of the second target 3D frame of the laser radar to be calibrated, so as to convert the coordinate system of the laser radar to be calibrated into the coordinate system of the target laser radar for subsequent matching analysis.

[0088] Step a42, calculating the average matching degree of all converted second target frames with the first target frame in the overlapping field of view of the target laser radar and the laser radar to be calibrated, and obtaining a matching score.

[0089] Determine the overlapping field of view angle range of the target LiDAR and the LiDAR to be calibrated, calculate the matching degree between the first target 3D frame and the converted second target 3D frame within this range, and average the matching degrees of all target 3D frames under all overlapping field of view angles to obtain the matching score.

[0090] Step a43, when the matching score is greater than a threshold or meets a termination condition, output the initial transformation moment as an initialization external parameter homogeneous matrix between the laser radar to be calibrated and the target laser radar.

[0091] If the calculated matching score is greater than the preset threshold, or other specified termination conditions are met, the current initial transformation matrix T is considered to be the required data, and it is output as the initialization external parameter homogeneous matrix between the laser radar to be calibrated and the target laser radar. Among them, the termination conditions include the number of iterations reaching the iteration upper limit / threshold, the transformation of the initial transformation matrix T is less than the threshold / minimum value, etc.

[0092] Step a44, when the matching score is less than or equal to the threshold, or the termination condition is not met, the initial transformation matrix is ​​adjusted, and the step of determining the initialized external parameter homogeneous matrix between the sensor to be calibrated and the target sensor by maximizing the mean matching degree of the first target frame and the second target frame is repeated until the above termination condition is met.

[0093] In this embodiment, compared with the traditional cascade calibration where multiple sensors are transferred and calibrated in sequence, resulting in gradual accumulation of errors, the calibration method based on the target 3D frame matching directly processes the overlapping field of view of the target laser radar and the laser radar to be calibrated, avoiding the error amplification caused by multiple transfers and accumulation like cascade calibration, and improving the accuracy of calibration. This method focuses on the overlapping area between two specific laser radars, and the calculation and optimization are more concentrated and targeted.

[0094] It should be noted that the final initialized extrinsic parameter homogeneous matrix can be an extrinsic parameter calibration between a laser radar to be calibrated and a target laser radar, or multiple laser radars to be calibrated can be processed in sequence by the same method to obtain an extrinsic parameter calibration between each laser radar to be calibrated and the target laser radar.

[0095] Based on the above first embodiment, a third embodiment of a multi-sensor joint calibration method is proposed. The third embodiment is used to illustrate a calibration method when the target sensor is a laser radar and the sensor to be calibrated is a millimeter-wave radar.

[0096] Specifically, the calibration process for each millimeter-wave radar to be calibrated is as follows:

[0097] Step b1, acquiring first point cloud data obtained after the target sensor perceives the target object and acquiring second point cloud data obtained after the sensor to be calibrated perceives the target object.

[0098] Step b2, convert the second point cloud data into the coordinate system of the target sensor through the initial transformation matrix T. The initial transformation T is obtained according to the sensor installation digital model, which contains detailed information about the installation position and posture of the millimeter-wave radar to be calibrated on the vehicle. Based on this, an initial transformation matrix T is preliminarily determined through mathematical calculation or a preset model. This initial transformation matrix T preliminarily describes the position and direction relationship of the millimeter-wave radar to be calibrated relative to the target laser radar, which needs to be optimized later.

[0099] In this step, the initial transformation matrix T is used to perform coordinate transformation on the second point cloud data of the millimeter-wave radar to be calibrated, so as to convert the coordinate system of the millimeter-wave radar to be calibrated into the coordinate system of the target lidar for subsequent matching analysis.

[0100] Step b3, based on the first point cloud data and the converted second point cloud data, matching the point cloud features of the same target object in the coordinate system where the target laser radar is located.

[0101] Step b4, if the matching degree of the point cloud features of the target object reaches the threshold or meets the termination condition, the initial transformation matrix T is determined to be the calibration external parameter matrix, that is, the initial transformation matrix is ​​output as the initialization external parameter homogeneous matrix between the millimeter wave radar to be calibrated and the target laser radar. Among them, the termination condition includes that the number of iterations reaches the iteration upper limit / threshold, the transformation of the initial transformation matrix T is less than the threshold / minimum value, etc.

[0102] Step b5, if the matching degree of the point cloud features of the target object does not reach the threshold or does not meet the termination condition, adjust the initial transformation matrix, and repeat the step of matching the point cloud features of the same target object in the coordinate system where the target lidar is located based on the first point cloud data and the converted second point cloud data until the termination condition is met.

[0103] In some embodiments, after the laser radar to be calibrated is calibrated, other sensors (such as other vehicle-mounted laser radars, millimeter-wave radars, cameras, etc.) establish a calibration coordinate system based on the target laser radar, and the point cloud data of other calibrated sensors (such as other vehicle-mounted laser radars, millimeter-wave radars, cameras, etc.) will be converted to the coordinate system of this target laser radar, integrating more point cloud information to increase the number of point clouds of characteristic targets in the calibration scene. That is, the point cloud of the target laser radar at this location can be converted to the target laser radar coordinate system with the calibrated point clouds of other laser radars and all the point clouds of the target laser radar to increase the number of point clouds of characteristic targets in the calibration scene.

[0104] Specifically, before step b3, the method further includes:

[0105] Step b6, point cloud fusion, for other calibrated lidars (the aforementioned lidars to be calibrated), use the calculated initialized extrinsic parameter homogeneous matrix to convert their point cloud data into the coordinate system of the target lidar, and fuse the converted point cloud with the point cloud of the target lidar to form a comprehensive point cloud data set containing more feature targets.

[0106] Based on the above-mentioned first embodiment, a fourth embodiment of a multi-sensor joint calibration method is proposed. The fourth embodiment is used to illustrate a calibration method when the target sensor is a lidar and the sensor to be calibrated is a camera.

[0107] Specifically, the calibration process for each camera to be calibrated is as follows:

[0108] Step c1, acquiring first point cloud data obtained after the target sensor perceives the target object and acquiring second point cloud data obtained after the sensor to be calibrated perceives the target object.

[0109] Step c2, mapping the first point cloud data to the coordinate system of the camera to be calibrated through the intrinsic parameter matrix and the extrinsic parameter matrix (R, T), the point cloud forms a contour representing the target object in the pixel space, and this contour is determined based on the distribution of the converted point cloud position in the image pixels, and is used for subsequent determination of the initialization extrinsic parameter homogeneous matrix.

[0110] Among them, the extrinsic parameter matrix is ​​used for the conversion from the target lidar coordinate system to the camera coordinate system to be calibrated, and the intrinsic parameter matrix is ​​used for the conversion from the camera coordinate system to the pixel coordinate system. The intrinsic parameter matrix is ​​determined according to the properties of the camera to be calibrated, including the focal length of the camera.

[0111] Specifically, converting the first point cloud data perceived by the target laser radar into the coordinate system of the camera to be calibrated includes:

[0112] 1) Conversion from world coordinate system to camera coordinate system.

[0113] The world coordinate system and the camera coordinate system are both in three-dimensional space, but the origin of the coordinates and the directions of the axes are different. Converting from the world coordinate system to the camera coordinate system requires a rigid body transformation, including rotation and translation. The rotation matrix R describes the rotation direction and angle of the coordinate axis of the world coordinate system relative to the coordinate axis of the camera coordinate system, and the translation matrix T describes the position of the origin of the world coordinate system in the camera coordinate system. The conversion is as follows:

[0114]

[0115] Among them, [X C Y C Z C 1] is the coordinate of the target object in the camera coordinate system, [X L Y L Z L 1] is the coordinate of the target object in the world coordinate system, (R is the 3x3 rotation matrix, T is the 3x1 translation matrix).

[0116] 2) Combination Figure 5 As shown, the transformation from the camera coordinate system to the image coordinate system.

[0117] The camera coordinate system is three-dimensional, while the image coordinate system is two-dimensional. This conversion is mainly based on the imaging principle of the camera, projecting the three-dimensional points in the camera coordinate system onto the image plane.

[0118] f is the focal length of the camera, and the point p(X C Y C Z C ) and the coordinate relationship of the projection point P(x,y) in the image coordinate system is:

[0119] The final conversion relationship is:

[0120]

[0121] 3) Conversion between image coordinate system and pixel coordinate system

[0122] The unit of the image coordinate system is usually millimeters, while the unit of the pixel coordinate system is pixels. This conversion needs to take into account the resolution of the image and the position of the image center in the pixel coordinate system.

[0123] The coordinates of the center point o of the image in the pixel coordinate system are (u0, v0), and the coordinates of the target object point in the image coordinate system are (x, y) and (u, v) in the pixel coordinate system. The transformation formula is: Where dx and dy are the length and width of a single pixel in the image plane, converted into a matrix:

[0124]

[0125] 4) The final coordinate transformation relationship between the laser radar and the camera is:

[0126]

[0127] As mentioned above, in the conversion process, a calibration tool is constructed to control the rotation and translation through the keyboard to map all the first point cloud data to the pixel space (image) through the external parameter matrix and the internal parameter matrix. Since the first point cloud data uses all target lidar point clouds, the contour data of the target object is guaranteed as much as possible in the three-dimensional space, that is, the target contour after mapping to the pixel space (image) is guaranteed. Finally, the matching of the target contour between the mapped point cloud and the image can be seen more clearly on the image, thereby improving the calibration accuracy.

[0128] Step c3: analyzing the matching status of the outline of the mapped first point cloud data and the outline of the second target frame.

[0129] After the first point cloud data is converted to pixel space, it is compared with the image taken by the camera. In the image, the overlap and difference between the contour formed by the point cloud and the contour of the actual target object in the image can be intuitively observed. Through this intuitive comparison, it is clear whether the current external parameter matrix (R and T) is accurate.

[0130] Step c4, when the matching result meets the termination condition, output the external parameter matrix as the initialized external parameter homogeneous matrix between the camera to be calibrated and the target laser radar.

[0131] When the target contour of the mapped point cloud completely matches the actual contour of the target object in the image, or reaches a preset high-precision matching level, the extrinsic parameter matrix used at this time is the final calibration amount, that is, the output extrinsic parameter matrix is ​​used as the initialization extrinsic parameter homogeneous matrix between the camera to be calibrated and the target lidar.

[0132] Step c5: When the matching result does not meet the termination condition (i.e., the target contour of the mapped point cloud does not match the actual contour of the target object in the image), adjust the rotation (R) and translation (T) through keyboard control. Continue to make subtle adjustments and observe the matching situation so that the point cloud contour coincides with the target contour in the image as much as possible. After each adjustment, re-convert and compare the coordinates, gradually optimize the calibration parameters, and thus improve the calibration accuracy.

[0133] In other embodiments, when calibrating each camera to be calibrated, the first target frame and the second target frame that are sensed by the point cloud data of the target laser radar and the camera to be calibrated are used for calibration.

[0134] Specifically, the calibration method of the fifth embodiment of the multi-sensor joint calibration method has the following calibration process for each camera to be calibrated:

[0135] Step d1, acquiring first point cloud data obtained after the target sensor perceives the target object and acquiring second point cloud data obtained after the sensor to be calibrated perceives the target object.

[0136] Step d2: identifying the target object in the acquired first point cloud data and the acquired second point cloud data respectively through a target detection algorithm.

[0137] Step d3, obtaining a first target frame obtained after the target sensor perceives the target object and a second target frame obtained after the sensor to be calibrated perceives the target object.

[0138] Specifically, bounding boxes are generated for these target objects based on the coordinates, size and orientation of each target object relative to the sensor, wherein the target laser radar generates a first target 3D box and the camera to be calibrated generates a second target 2D box.

[0139] Step d4: determining an initialized extrinsic parameter homogeneous matrix between the sensor to be calibrated and the target sensor according to a strategy for maximizing the mean value of the matching degree between the first target frame and the second target frame.

[0140] Specifically, the process includes the following steps:

[0141] Step d41, mapping the first target 3D frame sensed by the target laser radar to the coordinate system of the camera to be calibrated through the intrinsic parameter matrix and the extrinsic parameter matrix; the intrinsic parameter matrix is ​​determined according to the properties of the camera to be calibrated.

[0142] Step d42, matching and comparing the mapped target frame with the second target frame. Each target 2D frame mapped in the coordinate system of the camera to be calibrated by the first target 3D frame is matched with the second target 2D frame perceived by the camera to be calibrated, and the average matching degree of all converted second target 2D frames with the target 2D frame in the overlapping field of view of the target laser radar and the camera to be calibrated is calculated to obtain a matching score.

[0143] Step d43, when the matching result meets the termination condition, output the external parameter matrix as the initialized external parameter homogeneous matrix between the camera to be calibrated and the target laser radar.

[0144] If the calculated matching score is greater than the threshold or meets the termination condition, the current extrinsic parameter matrix is ​​considered to be the required data, the extrinsic parameter calibration is completed, and the current extrinsic parameter matrix is ​​output as the initialization extrinsic parameter homogeneous matrix between the camera to be calibrated and the target lidar. Among them, the termination conditions include the number of iterations reaching the iteration upper limit / threshold, the transformation of the initial transformation matrix T is less than the threshold / minimum value, etc.

[0145] Step d44, when the matching score is less than or equal to the threshold, or the termination condition is not met, the external parameter matrix is ​​adjusted, and the step of determining the initialized external parameter homogeneous matrix between the sensor to be calibrated and the target sensor by maximizing the mean matching degree of the first target frame and the second target frame is repeated until the termination condition is met.

[0146] In this embodiment, the matching degree of the target object is calculated based on the 2D frame obtained by mapping the 3D frame perceived by the target lidar to the coordinate system of the camera to be calibrated and the 2D frame perceived by the camera to be calibrated, so as to determine whether the target objects of the camera and the lidar are the same target by using an external parameter calibration method, which is beneficial to improving the accuracy of the post-target fusion algorithm in determining whether they are the same target.

[0147] Based on the same application concept as the above method, the present application embodiment also proposes a multi-sensor joint calibration device, such as Figure 6 shown.

[0148] The device comprises:

[0149] A target sensing module 602 is used to obtain a first target frame obtained after a target sensor senses a target object and a plurality of second target frames obtained after a plurality of sensors to be calibrated sense the target object respectively;

[0150] An initial calibration module 604 is used to determine an initialization extrinsic parameter homogeneous matrix between each of the sensors to be calibrated and the target sensor according to a strategy of maximizing the mean value of the matching degree between the first target frame and the second target frame;

[0151] The joint calibration module 606 is used to establish a joint calibration extrinsic parameter homogeneous matrix between the multiple sensors to be calibrated based on each of the initialized extrinsic parameter homogeneous matrices in a calibration coordinate system established based on the target sensor.

[0152] The implementation process of the functions and effects of each module / submodule / unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and the same technical effect can be achieved, so it will not be repeated here.

[0153] Figure 7 The following is a schematic diagram of the physical structure of a multi-sensor joint calibration device. Figure 7 As shown, the multi-sensor joint calibration device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the multi-sensor joint calibration method.

[0154] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0155] On the other hand, the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-sensor joint calibration method provided by the above methods.

[0156] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the multi-sensor joint calibration method provided by the above methods.

[0157] It should be noted that the technical solutions or technical features described in the above embodiments can be combined or supplemented with each other without causing conflicts. The scope of protection of this application is not limited to the precise structures described in the above embodiments and shown in the drawings; all modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A multi-sensor joint calibration method, characterized in that: The method comprises: Acquire a first target frame obtained after the target sensor perceives the target object and a plurality of second target frames obtained after a plurality of sensors to be calibrated perceive the target object respectively; Determining an initialized extrinsic parameter homogeneous matrix between each of the sensors to be calibrated and the target sensor according to a strategy for maximizing the mean value of the matching degree between the first target frame and the second target frame; In a calibration coordinate system established with the target sensor as a reference, a joint calibration extrinsic parameter homogeneous matrix between the multiple sensors to be calibrated is established based on each of the initialized extrinsic parameter homogeneous matrices.

2. The multi-sensor joint calibration method according to claim 1, characterized in that: In a calibration coordinate system established based on the target sensor, the point cloud data of the target object in the calibration coordinate system includes the point cloud data of the target object sensed by the target sensor and the point cloud data of the target object sensed by the sensor to be calibrated that has completed calibration and converted into the calibration coordinate system.

3. The multi-sensor joint calibration method according to claim 1, characterized in that: In the case where the target sensor is a target laser radar and the sensor to be calibrated is a laser radar to be calibrated, The step of determining an initialized extrinsic parameter homogeneous matrix between the sensor to be calibrated and the target sensor according to a strategy of maximizing the mean matching degree between the first target frame and the second target frame includes: Transforming the second target frame into the coordinate system of the target sensor through an initial transformation matrix; Calculate the average matching degree of all converted second target frames with the first target frame in the overlapping field of view of the target laser radar and the laser radar to be calibrated, and obtain a matching score; When the matching score meets the termination condition, the initial transformation matrix is ​​output as an initialized external parameter homogeneous matrix between the laser radar to be calibrated and the target laser radar.

4. The multi-sensor joint calibration method according to claim 3, characterized in that: When the matching score does not meet the termination condition, the initial transformation matrix is ​​adjusted, and the step of determining the initialized extrinsic parameter homogeneous matrix between the sensor to be calibrated and the target sensor by using the mean matching degree maximization strategy between the first target frame and the second target frame is repeated until the termination condition is met.

5. The multi-sensor joint calibration method according to claim 1, characterized in that: When the target sensor is a target laser radar and the sensor to be calibrated is a camera to be calibrated, The step of determining an initialized extrinsic parameter homogeneous matrix between the sensor to be calibrated and the target sensor according to a strategy of maximizing the mean matching degree between the first target frame and the second target frame includes: Mapping the first target frame to the coordinate system of the camera to be calibrated through an intrinsic parameter matrix and an extrinsic parameter matrix; the intrinsic parameter matrix is ​​determined according to the properties of the camera to be calibrated; Matching and comparing the mapped target frame with the second target frame; When the matching result meets the termination condition, the external parameter matrix is ​​output as the initialized external parameter homogeneous matrix between the camera to be calibrated and the target laser radar.

6. The multi-sensor joint calibration method according to claim 5, characterized in that: Before acquiring a first target frame obtained after the target sensor perceives the target object and a plurality of second target frames obtained after a plurality of sensors to be calibrated perceive the target object respectively, the method further comprises: The first point cloud data obtained after the target sensor senses the target object and the second point cloud data obtained after the sensor to be calibrated senses the target object are obtained.

7. The multi-sensor joint calibration method according to claim 6, characterized in that: The method further comprises: Mapping the first point cloud data to the coordinate system of the camera to be calibrated through an intrinsic parameter matrix and an extrinsic parameter matrix; Analyzing the matching between the outline of the mapped first point cloud data and the outline of the second target frame; When the matching result meets the termination condition, the external parameter matrix is ​​output as the initialized external parameter homogeneous matrix between the camera to be calibrated and the target laser radar.

8. The multi-sensor joint calibration method according to claim 6, characterized in that: In the case where the target sensor is a target laser radar and the sensor to be calibrated is a millimeter wave radar to be calibrated, The method further comprises: Feature matching is performed on the first point cloud data and the second point cloud data to determine an initialized extrinsic parameter homogeneous matrix between the target laser radar and the millimeter-wave radar to be calibrated.

9. A multi-sensor joint calibration device, characterized in that: It comprises a memory, a processor and a multi-sensor joint calibration program stored in the memory and executable on the processor, wherein the processor implements the steps of the multi-sensor joint calibration method as described in any one of claims 1 to 8 when executing the multi-sensor joint calibration program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multi-sensor joint calibration program, which, when executed, implements the steps of the multi-sensor joint calibration method according to any one of claims 1 to 8.

Images