Fusion calibration method for point cloud data and infrared data

Abstract

The invention discloses a point cloud data and infrared data fusion calibration method, and the method comprises the steps: introducing a calibrated color camera to assist the preprocessing of original sample data, enabling the color camera to play a crucial role in the fusion calibration, capturing a color image of an infrared checkerboard in a visible light environment, and carrying out the fusion calibration. According to the method, point cloud data collected by a laser radar are jointly used as input, infrared checkerboard angular point extraction, screening of unqualified infrared image data and filtering of redundant noise such as frame point cloud and environment point cloud are achieved in cooperation with an algorithm, and finally independent infrared checkerboard point cloud extraction is achieved and corresponds to infrared images in a one-to-one mode. And efficient and accurate fusion calibration of the point cloud data and the infrared data is realized.

Description

Technical Field

[0001] The invention belongs to the technical field of joint calibration, and in particular relates to a fusion calibration method of point cloud data and infrared data. Background Art

[0002] Due to the inherent limitations of a single sensor system, it is difficult to ensure the accuracy of perception in a changing environment. Therefore, the current machine vision field mostly uses multi-sensor collaboration to meet actual needs. For example, the combination of infrared cameras and lidar can significantly enhance environmental perception and data analysis capabilities, especially in the fields of autonomous driving, robot navigation, and environmental monitoring. Infrared cameras can capture thermal radiation images in low light or at night, are sensitive to temperature changes, and can identify heat sources and thermal contrast; lidar generates high-precision 3D environmental models through laser scanning, providing accurate depth information and object distance. The depth information provided by lidar combined with the thermal image of the infrared camera can achieve more accurate classification and identification of the environment and objects. Compared with the collaborative work of visible light cameras and lidar, the joint solution of lidar and infrared cameras can avoid the interference of darkness and glare environments, and has greater advantages in identifying objects at night. Compared with infrared cameras, the collaborative work of lidar and infrared cameras can more accurately identify the distance, posture, category and location information of pedestrians and vehicles in front of the vehicle, so that smart cars can also have better environmental perception capabilities at night.

[0003] However, accurate calibration of infrared cameras and lidars is a prerequisite for ensuring spatial consistency of data. Accurate calibration of infrared cameras and lidars is a prerequisite for ensuring spatial consistency of data. The checkerboard calibration method is a mature camera calibration and image correction technology that calculates the internal and external parameters of the camera by taking images on a known pattern (such as an infrared checkerboard). The joint calibration of lidar and infrared cameras using the infrared checkerboard method is to collect data on the infrared checkerboard with the infrared camera, so that each frame of the image captured by the infrared camera corresponds to the point cloud data collected by the lidar. The internal and external parameters of the camera are then solved to complete the joint calibration.

[0004] At present, the joint calibration of infrared cameras and lidar mainly has the following problems:

[0005] 1. Infrared cameras obtain object information by detecting infrared radiation. In order to ensure that the infrared checkerboard has stable radiation characteristics in the infrared band and improve the accuracy of calibration, heating elements are embedded in the infrared checkerboard to ensure that the infrared checkerboard can maintain a uniform temperature after heating. These heating elements need to be powered to heat and start, so the use of wires is inevitable. These heating elements may cause the laser radar to use part of the point cloud noise (the "tail" wires connected to the checkerboard) as redundant point cloud noise when performing point cloud recognition, which increases the difficulty of calibration and the error rate.

[0006] 2. The infrared camera will be affected by the reflective factor of the infrared chessboard surface material, which affects the corner extraction and recognition. Therefore, the data will be collected at different distances and postures in order to solve this problem to a certain extent. However, the laser radar has not solved the problem that the point cloud effect is abnormally sparse due to the reflective surface of the infrared chessboard.

[0007] 3. The borders around the standard infrared chessboard will also bring certain errors. If you want to improve the calibration efficiency by making your own infrared chessboard, first of all, the operation of making your own infrared chessboard itself involves a certain amount of work and difficulty, and it cannot be guaranteed to be more reliable than the market standard infrared chessboard, and it cannot guarantee better calibration efficiency. Summary of the invention

[0008] In view of the shortcomings of the existing joint calibration of infrared cameras and lidars, the present invention proposes a fusion calibration method of point cloud data and infrared data, which introduces a color camera with calibrated internal parameters and lidar external parameters to assist in preprocessing the original sample data, and uses color image data as the judgment standard to eliminate all the infrared image data and point cloud data in the same frame corresponding to the unqualified color image data from which corner points cannot be extracted in the original sample data, so as to improve the calibration accuracy.

[0009] A fusion calibration method for point cloud data and infrared data is proposed. An infrared camera and a laser radar are rigidly fixed and infrared data and point cloud data are collected synchronously at different distances and postures. At the same time, a color camera with calibrated internal parameters and external parameters of the laser radar is used to follow the infrared camera in a visible light environment to synchronously collect color image data at different distances and postures in the same frame.

[0010] Here, the synchronous collection of infrared data and point cloud data means ensuring that the viewing angles and coordinate systems of the infrared camera and the laser radar are aligned, and using a synchronization trigger or software to synchronize the collection time of the two to ensure data consistency. The color camera is not limited to the binding combination between the infrared camera and the laser radar, and only needs to be mounted on any bracket or fixed somewhere for auxiliary collection.

[0011] The color image data and the point cloud data collected by the lidar in the same frame are taken as input, and the corner point extraction is realized through feature extraction. Based on the calibration results of the color camera and lidar, the collected point cloud is filtered, and the point cloud data of the chessboard itself is retained. Taking the color image data as the standard, the unqualified color image data in the original sample data, in which the corner points cannot be extracted, and the corresponding infrared image data and point cloud data in the same frame are all eliminated.

[0012] The preprocessed color image data from which corner points can be extracted is used to correspond to the infrared image data collected in the same frame, and the intrinsic parameter matrix value of the infrared camera is calculated.

[0013] Using the preprocessed infrared image data and the filtered extracted infrared chessboard point cloud data, the rigid transformation from the lidar to the infrared camera is estimated, and finally the external parameter matrix value is obtained, and the fusion calibration is completed.

[0014] Preferably, when the laser radar collects point cloud data, a non-reflective film of the same size as the infrared checkerboard is used to cover the infrared checkerboard.

[0015] The beneficial effects of the present invention include: introducing a calibrated color camera to assist in preprocessing the original sample data, the color camera plays a vital role in fusion calibration, capturing the color image of the infrared checkerboard under the visible light environment, and coordinating the point cloud data collected by the laser radar as input, and coordinating the algorithm to realize the extraction of infrared checkerboard corner points, screen out unqualified infrared image data, and filter out redundant noise such as border point cloud and environmental point cloud, and finally realize the extraction of separate infrared checkerboard point cloud, and correspond one to one with the infrared image; using a non-reflective film of the same size as the infrared checkerboard as a mask layer, blocking the infrared checkerboard plane, perfectly solving the problem of abnormally sparse point cloud when the laser radar directly collects the infrared checkerboard; introducing a calibrated color camera to assist in preprocessing the original sample data, combined with using a non-reflective film of the same size as the infrared checkerboard as a mask layer, realizing efficient and accurate fusion calibration of point cloud data and infrared data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the method for fusion calibration of point cloud data and infrared data disclosed in the present invention;

[0017] Figure 2 This is the point cloud data before the non-reflective film is blocked;

[0018] Figure 3 This is the point cloud data after being blocked by non-reflective film;

[0019] Figure 4 This is the sample data before point cloud filtering;

[0020] Figure 5It is the sample data after point cloud filtering;

[0021] Figure 6 This is the performance test result of the point cloud data and infrared data fusion calibration method disclosed in the present invention. DETAILED DESCRIPTION

[0022] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are provided for the purpose of illustration and description, and are not intended to be exhaustive or to limit the present invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention, and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

[0023] Example 1

[0024] A fusion calibration method of point cloud data and infrared data, such as Figure 1 As shown, the infrared camera and lidar are used to synchronously collect infrared data and point cloud data at different distances and postures. At the same time, a calibrated color camera is used to follow the infrared camera in a visible light environment to synchronously collect color image data at different distances and postures in the same frame.

[0025] When the laser radar collects point cloud data, a non-reflective film of the same size as the infrared chessboard is used to block the infrared chessboard. Figure 2 , Figure 3 As shown in the figure, before occlusion, the chessboard point cloud cannot be fully displayed due to factors such as the reflection of the chessboard surface. Figure 2 ; After blocking, the reflection problem is solved and the point cloud is clearly visible. Figure 3 .

[0026] Here, the synchronous collection of infrared data and point cloud data means ensuring that the viewing angles and coordinate systems of the infrared camera and the laser radar are aligned, and using a synchronization trigger or software to synchronize the collection time of the two to ensure data consistency. The color camera is not limited to the binding combination between the infrared camera and the laser radar, and only needs to be mounted on any bracket or fixed somewhere for auxiliary collection.

[0027] The color image data and the point cloud data collected by the lidar in the same frame are used as input, and the findChessboardCornersSB function under the OpenCV library is used to find corner points with sub-pixel accuracy, the drawChessboardCorners function is used to extract corner points, and the projectPoints function is used for projection to achieve environmental point cloud filtering. OpenCV is a widely used open source computer vision and image processing library that provides a large number of library functions.

[0028] Taking color image data as the standard, all the unqualified color image data whose corner points cannot be extracted in the original sample data and the corresponding infrared image data and point cloud data in the same frame are eliminated. Figure 4 , Figure 5 The sample data before and after point cloud filtering are given for comparison. It is obvious that after point cloud filtering, Figure 5 The goal is clear and most of the interference factors are filtered out.

[0029] The infrared image data collected in the same frame is used to calculate the intrinsic parameter matrix value of the infrared camera by using the preprocessed color image data that can extract the corner points; the rigid transformation from the laser radar to the infrared camera is estimated by using the preprocessed infrared image data and the point cloud data of the infrared chessboard itself after filtering and extraction, and finally the extrinsic parameter matrix value is obtained, and the fusion calibration is completed. This part belongs to the prior art and will not be described in detail here.

[0030] The following is a performance test of the point cloud data and infrared data fusion calibration method disclosed in this embodiment. The test results are as follows: Figure 6 As shown in the bar chart on the lower right, the average error is 1.5 pixels and the maximum error is 3.5 pixels, which meets the calibration error requirements.

[0031] In summary, the present invention not only improves the work efficiency of calibration, but also improves the accuracy, that is, reduces the error rate of direct acquisition, based on the good solution of the calibration difficulty problem. At the same time, the present invention not only does not need to customize the non-standard infrared chessboard, which greatly reduces the cost, but also the position and environment of the color camera used for auxiliary fusion calibration are not restricted, that is, it does not need to be combined and bound with the synchronous acquisition module (infrared camera + laser radar), and only needs to be fixed by any bracket to be able to collect data normally in a visible light environment, which greatly improves flexibility.

[0032] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field and related fields without creative work should fall within the scope of protection of the present invention.

Claims

1. A method for fusion calibration of point cloud data and infrared data, by rigidly fixing an infrared camera and a laser radar, and collecting infrared data and point cloud data synchronously at different distances and different postures, characterized in that: A color camera with calibrated internal parameters and external parameters of the lidar is used to synchronously collect color image data at different distances and postures in the same frame in a visible light environment following the infrared camera. The color image data and the point cloud data collected by the lidar in the same frame are used as input, and the corner point extraction is realized through feature extraction. The color image data is used as the standard, and the infrared image data and point cloud data in the same frame corresponding to the unqualified color image data in the original sample data that cannot extract the corner point are completely eliminated; Use the pre-processed color image data that can extract corner points to correspond to the infrared image data collected in the same frame, and calculate the intrinsic parameter matrix value of the infrared camera; Using the preprocessed infrared image data and point cloud data, the rigid transformation from the lidar to the infrared camera is estimated, and finally the external parameter matrix value is obtained, and the fusion calibration is completed.

2. The method for fusion calibration of point cloud data and infrared data according to claim 1, characterized in that: When the laser radar collects point cloud data, it uses a non-reflective film of the same size as the infrared checkerboard to cover the infrared checkerboard.

3. The method for fusion calibration of point cloud data and infrared data according to claim 1 or 2, characterized in that: Based on the calibration results of the color camera and lidar, the collected point cloud is filtered, and the point cloud data of the chessboard itself is retained. Then, based on the filtered point cloud data, the rigid transformation from the lidar to the infrared camera is estimated, and finally the extrinsic parameter matrix value is obtained.

Images