Space-time joint calibration method for visible light camera, infrared camera and laser radar

Abstract

The invention discloses a space-time joint calibration method for a visible light camera, an infrared camera and a laser radar, and relates to the technical field of joint calibration of multi-modal sensors. Aiming at the problem of reference non-uniformity existing in time-sharing calibration, a time synchronization reference with microsecond-level precision is established through combination of FPGA-based hardware synchronization triggering and software interpolation delay compensation, and high consistency of different modal data at a collection moment is ensured; aiming at the problem that direct calibration of an infrared camera and a laser radar is difficult, a multi-stage calibration strategy with a visible light camera as a medium is formulated, two extrinsic parameter matrixes are solved respectively, and a direct extrinsic parameter matrix between the infrared camera and the laser radar is obtained through coordinate transformation; for the problem of error accumulation of a multi-stage calibration strategy in an external parameter transmission process, a robust loss function is introduced to suppress the influence of residual external points, and a nonlinear optimization method is adopted to carry out iterative optimization solution on rotation and translation parameters.

Description

TECHNICAL FIELD

[0001] The spatio-temporal joint calibration method of the visible light camera, the infrared camera and the laser radar of the application relates to the technical field of multi-modal sensor joint calibration. BACKGROUND

[0002] In the current field of computer vision, single modal detection systems are still the mainstream choice for some scenes, but due to the limitations of the sensor's own detection mechanism, the adaptability and information integrity of single modal detection systems in complex scenes have significant defects. Specifically, visible light cameras rely on passive reflection of objects to form images, and in harsh weather or low light environments, images are prone to have a large number of noise points and target texture details are blurred. Although infrared cameras can capture target thermal radiation information, they cannot reflect the color and texture details of the scene, and cannot obtain the depth information of the target when used alone. Laser radar can collect three-dimensional point cloud data of the target and obtain depth information, but it cannot directly present the texture characteristics and thermal radiation characteristics of the target when used alone, and it is prone to measurement errors when used in close-range detection. The limitations of single modal detection systems make it difficult to meet the demand for simultaneous acquisition of multi-dimensional information in complex scenes in the fields of autonomous driving, military reconnaissance, etc.

[0003] Multi-modal data fusion of visible light, infrared and laser radar information can fully utilize the unique advantages of each sensor in high spatial resolution, long wave penetration characteristics and three-dimensional modeling, effectively solving the performance degradation problem of traditional technology in complex scenes such as low light and harsh weather, and achieving high-precision perception of target spatial pose, meeting the stringent requirements of image recognition and spatial pose in the fields of autonomous driving, intelligent security, etc.

[0004] In a multi-modal image fusion detection system, multi-modal image fusion methods can be divided into three categories according to the "data processing level": pixel-level fusion, feature-level fusion and decision-level fusion, and the methods of pixel-level fusion and feature-level fusion are based on image registration as a prerequisite.

[0005] Existing registration schemes are mostly for alignment of two modalities, and there is a lack of three-modal registration schemes for "visible light camera, infrared camera and laser radar", and the following limitations exist:

[0006] First, from the time dimension, visible light and infrared cameras rely on passive collection of environmental light reflection and thermal radiation of the target to form images, and laser radar relies on active emission of laser pulse echoes to obtain information. Due to different imaging mechanisms, the time response characteristics of data acquisition are different. The current three-modal registration scheme lacks high-precision time synchronization operation, and the misalignment in time will cause ghosting in motion scenes.

[0007] Second, from the spatial dimension, the three are collected in different hardware devices, with different spatial coordinate systems; the existing "visible light camera, infrared camera and laser radar" three modal registration scheme mainly has two kinds: using three kinds of sensors to take a shot of the calibration board once and to carry out synchronous calibration single frame calibration scheme, and through two-by-two registration and transmission of external parameters multi-level calibration scheme; wherein the former can directly obtain the spatial coordinate system corresponding relationship of three modal data, but is only used for static scene; the latter is robust, suitable for dynamic scene and complex environment, however, in such joint calibration process, the error will be accumulated with the transmission of external parameters, finally leading to the decline of registration accuracy.

[0008] Therefore, there is an urgent need in the art for a high-precision, high-robustness, suitable for dynamic scene and complex environment, visible light camera, infrared camera and laser radar spatio-temporal joint calibration method. SUMMARY

[0009] The present application provides a visible light camera, infrared camera and laser radar spatio-temporal joint calibration method, aiming at the problems of lack of three modal registration scheme, lack of high-precision time synchronization mechanism and significant error accumulation in the prior art, to provide a visible light camera, infrared camera and laser radar spatio-temporal joint calibration method to realize the unification of three modal data spatio-temporal reference, and lay the foundation for multi-modal image fusion algorithm.

[0010] The purpose of the present application is achieved as follows:

[0011] The visible light camera, infrared camera and laser radar spatio-temporal joint calibration method comprises the following steps:

[0012] Step a, hardware fixing and spatio-temporal reference establishment;

[0013] Step b, visible light camera and infrared camera internal parameter calibration;

[0014] Step c, cross-modal external parameter joint calibration;

[0015] Step d, calibration result verification.

[0016] The visible light camera, infrared camera and laser radar spatio-temporal joint calibration method described above, step a is specifically:

[0017] The visible light camera, infrared camera is fixed by a rigid support, and the laser radar is rigidly connected with the support, so as to ensure that the relative pose of the three is constant; after hardware fixing, the relative distance of the visible light camera, infrared camera and laser radar is measured, which is used as the reference for subsequent calibration result verification;

[0018] Achieving visible light camera, infrared camera and laser radar data acquisition time synchronization, the time synchronization includes FPGA-based hardware trigger and software time interpolation compensation; the FPGA-based hardware trigger combines with software time interpolation compensation to establish a unified time reference;

[0019] The FPGA-based hardware trigger control circuit includes an FPGA master chip, a clock input interface and a multi-channel trigger output circuit, for receiving an external synchronization clock signal and generating a multi-channel TTL level synchronization trigger signal to synchronize the visible light camera, infrared camera and laser radar to collect data;

[0020] The software time interpolation compensation uses linear or cubic spline interpolation algorithm to correct the data time difference of sensors with different acquisition frame rates.

[0021] The above-mentioned spatio-temporal joint calibration method of visible light camera, infrared camera and laser radar, step b is specifically:

[0022] A checkerboard calibration board is used;

[0023] The visible light camera and the infrared camera are controlled to respectively capture multiple groups of images of the checkerboard calibration board from different poses;

[0024] The visible light image is denoised, and the infrared image is subjected to non-uniformity correction and bilateral filtering;

[0025] Sub-pixel level coordinates of the checkerboard corner points are extracted from the preprocessed images;

[0026] A combination method of corner point preliminary screening and sub-pixel precise positioning is used to extract the corner point coordinates;

[0027] Images with a reprojection error greater than a preset threshold are removed, and the intrinsic matrix and distortion coefficient of the visible light camera and the infrared camera are respectively solved.

[0028] The above-mentioned spatio-temporal joint calibration method of visible light camera, infrared camera and laser radar, step c is specifically:

[0029] A multi-level calibration strategy is formulated with the visible light camera as an intermediary, based on the intrinsic matrix obtained in step b, the data of the checkerboard calibration board collected synchronously are used to respectively solve the extrinsic matrix of the infrared camera relative to the visible light camera and the extrinsic matrix of the laser radar relative to the visible light camera;

[0030] Robust estimation algorithm is used to remove abnormal samples to obtain initial extrinsic parameter estimation;

[0031] A weighted sum of reprojection errors of each inlier is taken as an objective function, a robust loss function is introduced to suppress the influence of residual outliers, and a nonlinear optimization method is used to iteratively optimize and solve the rotation and translation parameters;

[0032] Based on the extrinsic matrix of the infrared camera relative to the visible light camera and the extrinsic matrix of the lidar relative to the visible light camera, a direct extrinsic matrix between the infrared camera and the lidar is obtained through coordinate transformation.

[0033] The above-mentioned spatio-temporal joint calibration method of the visible light camera, the infrared camera and the lidar, step d is specifically:

[0034] Based on the extrinsic matrix of the infrared camera relative to the visible light camera and the extrinsic matrix of the lidar relative to the visible light camera, the infrared image and the lidar point cloud are projected to the visible light image.

[0035] Objective evaluation indexes are calculated for the aligned data to measure geometric consistency and registration accuracy.

[0036] Further, the fusion effect is judged by subjective visual evaluation.

[0037] Compared with the prior art, the beneficial effects of the spatio-temporal joint calibration method of the visible light camera, the infrared camera and the lidar of the present application are:

[0038] First, for the problem of inconsistent reference existing in time-sharing calibration, the present application establishes a time synchronization reference with microsecond level precision by combining hardware synchronization trigger based on FPGA with software interpolation delay compensation, ensuring high consistency of different modal data at the time of acquisition; at the same time, a stable and unified spatial coordinate system is constructed by fixing with a rigid support and taking the visible light camera as a unified spatial medium.

[0039] Second, for the problem that it is difficult to directly calibrate the infrared camera and the lidar, the present application formulates a multi-stage calibration strategy taking the visible light camera as an intermediate, uses the synchronously collected checkerboard calibration board data to respectively solve the extrinsic matrix of the infrared camera relative to the visible light camera and the extrinsic matrix of the lidar relative to the visible light camera, and obtains the direct extrinsic matrix between the infrared camera and the lidar through coordinate transformation.

[0040] Third, for the problem of error accumulation in the extrinsic parameter transmission process of the multi-stage calibration strategy, the present application eliminates abnormal samples by robust estimation algorithm to obtain initial extrinsic parameter estimation; takes the weighted sum of squares of re-projection errors of each inlier as the objective function, introduces a robust loss function to suppress the influence of residual outliers, and uses a nonlinear optimization method to iteratively optimize and solve the rotation and translation parameters; reduces the error accumulated in the process of introducing abnormal data and extrinsic parameter transmission in the corner detection process. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a flowchart of the spatio-temporal joint calibration method of the visible light camera, the infrared camera and the lidar of the present application;

[0042] Figure 2 is a schematic diagram of a spatio-temporal joint calibration system of visible light camera, infrared camera and lidar;

[0043] Figure 3 is a schematic diagram of a time registration method;

[0044] Figure 4 is a flow chart of cross-modality extrinsic parameter optimization;

[0045] Figure 5 is a schematic diagram of extrinsic parameter transfer and unified integration in a spatio-temporal joint calibration method. DETAILED DESCRIPTION

[0046] The specific embodiments of the application will be further described in detail below with reference to the accompanying drawings. Specific embodiment one

[0048] The spatio-temporal joint calibration method of visible light camera, infrared camera and lidar under this specific embodiment has a flow chart as shown in Figure 1 , which includes the following steps:

[0049] Step a, hardware fixation and spatio-temporal reference establishment;

[0050] Step b, visible light camera and infrared camera intrinsic parameter calibration;

[0051] Step c, cross-modality extrinsic parameter joint calibration;

[0052] Step d, calibration result verification. Specific embodiment two

[0054] The spatio-temporal joint calibration method of visible light camera, infrared camera and lidar under this specific embodiment, on the basis of specific embodiment one, further limits that step a is specifically:

[0055] An installation base with rigid support capability is set, and a unified positioning reference surface and reference edge are formed, a standardized installation interface is provided on the base for constraint positioning and adjustment of the visible light camera, infrared camera and lidar, as shown in Figure 2 ;

[0056] Reading the original data time stamp of each sensor;

[0057] Taking the measurement time sequence of lidar point cloud as the reference time axis;

[0058] Performing time interpolation calculation on the image data of infrared camera or visible light camera to obtain a virtual image frame aligned with the radar measurement time;

[0059] Generating time-aligned multi-modality data sequence for subsequent extrinsic parameter calibration and fusion;

[0060] In this embodiment, to achieve inter-frame alignment, the application adopts a linear interpolation algorithm to calculate the intermediate time position between adjacent sampling points, and the schematic diagram of the time registration method is shown in Figure 3 ;

[0061] Taking the measurement time frame of the laser radar point cloud as the reference, let the radar frame target alignment time be T l , the corresponding front and rear infrared image times be T i_prev and T i_next , the front and rear time frame infrared image corresponding luminance values at (x, y) be I prev (x, y) and I next (x, y), then the luminance value I sync (x, y) of the virtual infrared image at the pixel (x, y) corresponding to the radar frame alignment time T l is as follows:

[0062]

[0063] This method is suitable for scenes with relatively uniform sampling intervals and small jitter;

[0064] In another embodiment of the application, to improve the accuracy of time interpolation, the application can adopt a cubic spline interpolation algorithm;

[0065] Let the four known adjacent time stamps be T and the corresponding pixel luminance values be I , , , , then the interpolation luminance value I at any intermediate time T can be expressed as:

[0066]

[0067] Wherein, the coefficient is determined by the time and luminance value of the adjacent sampling points, and satisfies the following conditions:

[0068]

[0069] Compared with linear interpolation, cubic spline interpolation performs better in scenes with inconsistent frame rates or sampling jitter. Embodiment Three

[0071] The space-time joint calibration method of the visible light camera, infrared camera and laser radar in this embodiment, on the basis of embodiment one, further limits step b to be:

[0072] Using a checkerboard calibration board;

[0073] The calibration board is placed in different postures and distances, and multiple groups of images are collected by controlling the visible light camera and the infrared camera to be triggered synchronously to ensure that the corner points in the images are uniformly distributed and cover the imaging area;

[0074] To reduce noise interference and improve the clarity of the corner point features, the visible light image is subjected to smoothing filter denoising, and the infrared image is subjected to non-uniformity correction and smoothing processing.

[0075] For the preprocessed image, a sub-pixel level corner point detection algorithm is used to extract the checkerboard corner points. In multiple groups of images, if the corner point recognition is incomplete or the repetition error is large, the effective image is automatically removed and used for solving.

[0076] Based on the world coordinates and pixel coordinates of the corner points, the internal parameter matrix of the visible light camera and the infrared camera is solved by using Zhang Zhengyou calibration algorithm according to the corresponding relationship between the world coordinates of the corner points and the image pixel coordinates. and the distortion coefficient .

[0077] The obtained camera internal parameter is used for image back projection to verify the consistency of the corner points in the re-projection coordinates, and the abnormal image data with a re-projection error greater than a preset threshold is removed.

[0078] The "sub-pixel level corner point detection algorithm" is a known technology in the art, which aims to improve the pixel level corner point coordinate accuracy to sub-pixel level, thereby significantly improving the accuracy of camera calibration. The specific implementation modes include but are not limited to:

[0079] Gradient-based method: using the gradient information of the corner point neighborhood, the best corner point position is found by solving the least squares problem;

[0080] Eigenvalue analysis-based method: first, use the Harris corner point detection algorithm or Shi-Tomasi algorithm to preliminarily screen and locate the corner points, and then use the above gradient-based method or other iterative method in the pixel neighborhood to refine the sub-pixel;

[0081] Template matching-based method: the corner point template is correlated with the image region to accurately locate;

[0082] In a preferred embodiment of the present application, the following process is used: first, use the Harris algorithm to quickly locate the pixel-level approximate position of the corner point, and use the sub-pixel corner point optimization algorithm based on gradient iterative convergence to finely process the preliminarily screened corner points at the sub-pixel level, and a good balance between efficiency and accuracy is achieved. DETAILED DESCRIPTION FOUR

[0084] The spatio-temporal joint calibration method of the visible light camera, the infrared camera and the lidar in the specific embodiment is based on the specific embodiment one, and further limits step c to be:

[0085] The data collected by the visible light camera, the infrared camera and the lidar are preprocessed.

[0086] Based on the calibration board data pairs of the same pose collected by the visible light camera and the infrared camera and the visible light camera and the lidar and the respective intrinsic parameters, initial extrinsic parameter estimations of the infrared camera and the lidar relative to the visible light camera are calculated.

[0087] The multiple sets of initial extrinsic parameter estimations calculated are taken as initial values, an outlier rejection algorithm is used to reject outliers, a nonlinear optimization algorithm is used for optimization with the objective of minimizing the re-projection error, and a final high-precision extrinsic parameter matrix is obtained. Specific embodiment five

[0089] The spatio-temporal joint calibration method of the visible light camera, the infrared camera and the lidar in the specific embodiment is based on the specific embodiment four, and further limits that the preprocessing of the data preferably uses the following in the specific embodiment:

[0090] For the visible light image: a lens distortion correction method based on a geometric model is used to correct the image, and Gaussian noise is removed through Gaussian filtering;

[0091] For the infrared image: a two-point correction method is used to eliminate non-uniformity noise of the detector caused by lens stray light, and a bilateral filter is used to remove thermal noise and retain edge information;

[0092] For the lidar point cloud: first, environmental interference points are removed through statistical outlier filtering, and then voxelization is performed on the point cloud data to complete spatial division of the point cloud data for subsequent processing. Specific embodiment six

[0094] The spatio-temporal joint calibration method of the visible light camera, the infrared camera and the lidar in the specific embodiment is based on the specific embodiment four, and further limits that in solving the relative extrinsic parameter of the infrared camera to the visible light camera, since errors may be introduced in the image pair collection, corner point detection and other links, directly calculating the average value of the extrinsic parameters obtained from all image pairs will lead to a decrease in calibration accuracy, and therefore the present application uses an outlier rejection algorithm and a nonlinear optimization algorithm for optimization, as shown in Figure 4 The specific steps are as follows:

[0095] Outlier rejection: Take one relative pose hypothesis calculated from each group of image pairs as a data sample, and use the RANSAC algorithm to iteratively randomly sample and verify, calculate the norm distance between the two rotation matrices and the Euclidean distance between the two translation vectors, and select the highest consistency of the inlier point set to obtain the initial extrinsic parameter estimation result;

[0096] Nonlinear optimization: In order to further improve the calibration accuracy, the extrinsic parameter estimation value obtained by the above RANSAC process is used as the initial value of nonlinear optimization; the optimization objective is to minimize the sum of the re-projection errors of all inlier image pairs, and the Levenberg-Marquardt nonlinear optimization algorithm is used to iteratively optimize the rotation matrix and translation vector, and finally the optimal extrinsic parameter solution with high precision is obtained;

[0097] This method effectively overcomes the influence of single data error and mismatching on the calibration result, and significantly improves the accuracy and robustness of the extrinsic parameter calibration. DETAILED DESCRIPTION SEVEN

[0099] The spatio-temporal joint calibration method of the visible light camera, the infrared camera and the lidar in this embodiment, based on the specific embodiment one, further limits step d to be:

[0100] The method of selecting feature points is used to transform the infrared image pixels to the visible light image pixels;

[0101] The lidar point cloud is projected to the visible light image;

[0102] The image fusion algorithm is used to realize the fusion of the visible light image and the infrared image, and the fusion of the visible light image and the lidar point cloud, to generate a multi-modal fusion image;

[0103] The calibration accuracy is evaluated by calculating the re-projection error and performing subjective visual evaluation; the Euclidean distance between the pixel coordinates of the infrared image feature points projected to the visible light image and the corresponding feature points in the visible light image is calculated as the re-projection error of the infrared modal; the Euclidean distance between the pixel coordinates of the calibration board corner points in the lidar point cloud projected to the visible light image and the corresponding corner points in the visible light image is calculated as the re-projection error of the lidar modal.

[0104] Finally, the direct extrinsic parameter matrix between the infrared camera and the lidar is calculated through the extrinsic parameter transfer relationship as shown in Figure 5 , which is one of the final spatial output results of this joint calibration method.

[0105] It should be noted that the above only illustrates the specific embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0106] It should also be noted that the technical features listed in the above specific embodiments can be arranged and combined, and those skilled in the art can exhaust all the results after each arrangement and combination. All the results after arrangement and combination should be understood as disclosed by the present application.

Claims

1. A spatio-temporal joint calibration method of a visible light camera, an infrared camera and a lidar, characterized in that, The method comprises the following steps: Step a, hardware fixation and space-time reference establishment; Step b, visible light camera and infrared camera internal parameter calibration; Step c, cross-modal external parameter joint calibration; Step d, calibration result verification.

2. The spatio-temporal joint calibration method of visible light camera, infrared camera and lidar according to claim 1, characterized in that, Step a is specifically: The visible light camera, the infrared camera and the laser radar are fixed through a rigid support, and the laser radar is rigidly connected with the support to ensure that the relative poses of the three are constant. After hardware fixation, the relative distances of the visible light camera, the infrared camera and the laser radar are measured as the reference for subsequent calibration result verification; Time synchronization of visible light camera, infrared camera and laser radar data acquisition is realized, which includes FPGA-based hardware triggering and software time interpolation compensation; the FPGA-based hardware triggering combines with the software time interpolation compensation to establish a unified time reference; The FPGA-based hardware triggering control circuit includes an FPGA master control chip, a clock input interface and a multi-channel trigger output circuit, which is used for receiving an external synchronous clock signal and generating a plurality of TTL level synchronous trigger signals to synchronously trigger the visible light camera, the infrared camera and the laser radar to collect data; The software time interpolation compensation adopts a linear or cubic spline interpolation algorithm to correct the data time difference of sensors with different acquisition frame rates.

3. The spatio-temporal joint calibration method of visible light camera, infrared camera and lidar according to claim 1, characterized in that, Step b is specifically: A checkerboard calibration board is used; A plurality of images of the checkerboard calibration board are captured from different poses by the visible light camera and the infrared camera respectively; The visible light image is denoised, and the infrared image is subjected to non-uniformity correction and bilateral filtering; Sub-pixel level coordinates of the checkerboard corner points are extracted from the preprocessed images; A combination method of corner point preliminary screening and sub-pixel precise positioning is used to extract the corner point coordinates; Images with a re-projection error greater than a preset threshold are removed, and the internal parameter matrix and distortion coefficient of the visible light camera and the infrared camera are respectively solved.

4. The spatio-temporal joint calibration method of visible light camera, infrared camera and lidar according to claim 1, characterized in that, Step c is specifically: A multi-stage calibration strategy with the visible light camera as the intermediary is formulated, and based on the internal parameter matrix obtained in step b, the external parameter matrix of the infrared camera relative to the visible light camera and the external parameter matrix of the laser radar relative to the visible light camera are respectively solved by using the synchronously collected checkerboard calibration board data; Robust estimation algorithm is used to remove abnormal samples to obtain initial external parameter estimation; A weighted sum of re-projection errors of each inlier is taken as an objective function, a robust loss function is introduced to suppress the influence of residual outliers, and a nonlinear optimization method is used to iteratively optimize and solve the rotation and translation parameters; Based on the external parameter matrix of the infrared camera relative to the visible light camera and the external parameter matrix of the laser radar relative to the visible light camera, the direct external parameter matrix between the infrared camera and the laser radar is obtained through coordinate transformation.

5. The spatio-temporal joint calibration method of visible light camera, infrared camera and lidar according to claim 1, characterized in that, Step d is specifically: Based on the external parameter matrix of the infrared camera relative to the visible light camera and the external parameter matrix of the laser radar relative to the visible light camera, the infrared image and the laser radar point cloud are projected to the visible light image; Objective evaluation indexes are calculated for the aligned data to measure geometric consistency and registration accuracy.

6. The spatio-temporal joint calibration method of visible light camera, infrared camera and lidar according to claim 5, characterized in that, The fusion effect is judged by subjective visual evaluation.

Images