A calibration method for fusing point cloud data and infrared data

Abstract

The present invention discloses a method for fusion calibration of point cloud data and infrared data. A calibrated color camera is introduced to assist in preprocessing original sample data. The color camera plays a vital role in fusion calibration, captures a color image of an infrared checkerboard in a visible light environment, and uses the point cloud data collected by a laser radar as input. The algorithm is used to extract infrared checkerboard corner points, screen out unqualified infrared image data, and filter out redundant noise such as border point clouds and environmental point clouds. Finally, a separate infrared checkerboard point cloud is extracted, and the point cloud is corresponding to the infrared image one by one, so as to realize efficient and accurate fusion calibration of point cloud data and infrared data.

Description

Technical Field

[0001] The present invention belongs to the technical field of joint calibration, and particularly relates to a method for fusing and calibrating point cloud data and infrared data. Background Art

[0002] Due to the limitations of a single sensor system, it is difficult to ensure the accuracy of perception in a changing environment. Therefore, in the current field of machine vision, multi-sensor collaboration is mostly used to meet actual needs. For example, the combination of an infrared camera and a lidar can significantly enhance the environmental perception and data analysis capabilities, especially in the fields of autonomous driving, robot navigation, environmental monitoring, etc. The infrared camera can capture thermal radiation images under low light or night conditions, is sensitive to temperature changes, and can identify heat sources and thermal contrasts; the lidar generates a high-precision 3D environmental model through laser scanning, providing accurate depth information and object distances. The combination of the depth information provided by the lidar and the thermal image of the infrared camera can achieve more accurate classification and recognition of the environment and objects. Compared with the collaborative work of a visible light camera and a lidar, the joint scheme of a lidar and an infrared camera can avoid the interference of night and glare environments and has greater advantages in identifying objects at night. Compared with an infrared camera, the collaborative work of a lidar and an infrared camera can more accurately identify the pedestrians in front of the vehicle, the distance, attitude, category, and position information of the vehicle, enabling intelligent vehicles to also have good environmental perception capabilities at night.

[0003] However, accurate calibration of the infrared camera and the lidar is a prerequisite for ensuring the spatial consistency of data. Accurate calibration of the infrared camera and the lidar is a prerequisite for ensuring the spatial consistency of data. The checkerboard calibration method is a mature technology for camera calibration and image correction. By taking images on a known pattern (such as an infrared checkerboard), the internal and external parameters of the camera are calculated. For the joint calibration of the lidar and the infrared camera by the infrared checkerboard method, the infrared camera and the lidar are used together to collect data on the infrared checkerboard, so that each frame of the image captured by the infrared camera corresponds one-to-one with the point cloud data collected by the lidar. Then, the internal and external parameters of the camera are solved, and thus the joint calibration work is completed.

[0004] Currently, the following problems mainly exist in the joint calibration of the infrared camera and the lidar:

[0005] 1. An infrared camera obtains object information by detecting infrared radiation. To ensure that the infrared checkerboard has stable radiation characteristics in the infrared band and improve the accuracy of calibration, heating elements are embedded in the infrared checkerboard, and it is ensured that the infrared checkerboard can maintain a uniform temperature after heating. These heating elements need to be powered to start heating, so wires are inevitably used. When the lidar performs point cloud recognition, some of these heating elements may be taken as redundant point cloud noise (the "tail" wires connected to the checkerboard), which instead increases the calibration difficulty and error rate.

[0006] 2. The infrared camera is affected by the reflection of the surface material of the infrared checkerboard, which affects the corner extraction and recognition. Therefore, during acquisition, different distances and poses are used in an attempt to solve this problem to a certain extent. However, the abnormal sparsity of the point cloud effect caused by the reflection of the surface of the infrared checkerboard by the lidar has not been solved.

[0007] 3. The border existing around the standard infrared checkerboard will also bring certain errors. If a self-made infrared checkerboard is used to improve the calibration efficiency, first, the operation of making a self-made infrared checkerboard itself has a certain amount of work and difficulty, and it cannot be ensured to be more reliable than the market standard infrared checkerboard, nor can it guarantee better calibration efficiency. Summary of the Invention

[0008] In view of the deficiencies existing in the joint calibration of existing infrared cameras and lidars, the present invention proposes a fusion calibration method for point cloud data and infrared data. A color camera with calibrated internal parameters and lidar external parameters is introduced to assist in preprocessing the original sample data. Taking the color image data as the judgment standard, all the unqualified color image data in the original sample data that cannot extract corners, as well as the corresponding infrared image data and point cloud data in the same frame, are eliminated to improve the calibration accuracy.

[0009] A fusion calibration method for point cloud data and infrared data. The infrared camera and the lidar are rigidly fixed, and infrared data acquisition and point cloud data acquisition are synchronously performed at different distances and poses. At the same time, a color camera with calibrated internal parameters and lidar external parameters is used to synchronously collect color image data at different distances and poses in the same frame following the infrared camera in the visible light environment.

[0010] Here, the synchronous acquisition of infrared data and point cloud data means ensuring that the viewing angles and coordinate systems of the infrared camera and the lidar are aligned, using a synchronous trigger or software to synchronize the acquisition times of the two to ensure data consistency. The color camera is not limited by the binding combination between the infrared camera and the lidar, and only needs to be assisted in acquisition with any bracket or fixed somewhere.

[0011] The color image data and the point cloud data collected by the lidar in the same frame are taken as input, and the corner point extraction is realized through feature extraction. Based on the calibration results of the color camera and lidar, the collected point cloud is filtered, and the point cloud data of the chessboard itself is retained. Taking the color image data as the standard, the unqualified color image data in the original sample data, in which the corner points cannot be extracted, and the corresponding infrared image data and point cloud data in the same frame are all eliminated.

[0012] The preprocessed color image data from which corner points can be extracted is used to correspond to the infrared image data collected in the same frame, and the intrinsic parameter matrix value of the infrared camera is calculated.

[0013] Using the preprocessed infrared image data and the filtered extracted infrared chessboard point cloud data, the rigid transformation from the lidar to the infrared camera is estimated, and finally the external parameter matrix value is obtained, and the fusion calibration is completed.

[0014] Preferably, when the laser radar collects point cloud data, a non-reflective film of the same size as the infrared checkerboard is used to cover the infrared checkerboard.

[0015] The beneficial effects of the present invention include: introducing a calibrated color camera to assist in preprocessing the original sample data, the color camera plays a vital role in fusion calibration, capturing the color image of the infrared checkerboard under the visible light environment, and coordinating the point cloud data collected by the laser radar as input, and coordinating the algorithm to realize the extraction of infrared checkerboard corner points, screen out unqualified infrared image data, and filter out redundant noise such as border point cloud and environmental point cloud, and finally realize the extraction of separate infrared checkerboard point cloud, and correspond one to one with the infrared image; using a non-reflective film of the same size as the infrared checkerboard as a mask layer, blocking the infrared checkerboard plane, perfectly solving the problem of abnormally sparse point cloud when the laser radar directly collects the infrared checkerboard; introducing a calibrated color camera to assist in preprocessing the original sample data, combined with using a non-reflective film of the same size as the infrared checkerboard as a mask layer, realizing efficient and accurate fusion calibration of point cloud data and infrared data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the method for fusion calibration of point cloud data and infrared data disclosed in the present invention;

[0017] Figure 2 This is the point cloud data before the non-reflective film is blocked;

[0018] Figure 3 This is the point cloud data after being blocked by non-reflective film;

[0019] Figure 4 This is the sample data before point cloud filtering;

[0020] Figure 5The sample data after point cloud filtering;

[0021] Figure 6 The performance test results of the fusion calibration method for point cloud data and infrared data disclosed by the present invention. Specific embodiments

[0022] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are given for purposes of illustration and description, and are not exhaustive or limit the present invention to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described to better illustrate the principles of the present invention and its practical applications, and to enable those of ordinary skill in the art to understand the present invention and design various embodiments with various modifications suitable for specific purposes.

[0023] Embodiment 1

[0024] A fusion calibration method for point cloud data and infrared data, as Figure 1 shown, infrared data acquisition and point cloud data acquisition are respectively carried out by an infrared camera and a lidar at different distances and different poses, and at the same time, a calibrated color camera is used to synchronously acquire color image data at different distances and different poses in the same frame in a visible light environment following the infrared camera.

[0025] When the lidar acquires point cloud data, a non-reflective film of the same size as the infrared checkerboard is used to cover the infrared checkerboard. The comparison diagrams before and after the covering are as Figure 2 、 Figure 3 shown. Before the covering, due to factors such as the reflection on the surface of the checkerboard, the checkerboard point cloud cannot be completely displayed, referring to Figure 2 ; after the covering, the reflection reason is solved and the point cloud is clearly visible, referring to Figure 3 .

[0026] Here, the synchronous acquisition of infrared data and point cloud data means ensuring the alignment of the viewing angles and coordinate systems of the infrared camera and the lidar, and using a synchronous trigger or software to synchronize the acquisition times of the two to ensure the consistency of the data. The color camera is not limited by the binding combination between the infrared camera and the lidar, and only needs to be assisted by any bracket or fixed somewhere for acquisition.

[0027] Taking the color image data and the point cloud data collected by the lidar in the same frame as the input, using the findChessboardCornersSB function in the OpenCV library to find corners with sub-pixel accuracy, the drawChessboardCorners function to extract corners, and the projectPoints function to perform projection to achieve environmental point cloud filtering. OpenCV is a widely used open-source computer vision and image processing library that provides a large number of library functions.

[0028] Taking the color image data as the standard, all the unqualified color image data in the original sample data that cannot extract corners and the corresponding infrared image data and point cloud data in the same frame are eliminated. Figure 4 、 Figure 5 The sample data before and after point cloud filtering are compared respectively. Obviously, the Figure 5 target in it is clear, and most of the interference factors are filtered out.

[0029] Using the color image data that can extract corners after preprocessing and the corresponding infrared image data collected in the same frame, the internal parameter matrix value of the infrared camera is calculated; using the preprocessed infrared image data and the point cloud data that only retain the infrared checkerboard itself after filtering and extraction, the rigid transformation from the lidar to the infrared camera is estimated, and finally the external parameter matrix value is obtained. Thus, the fusion calibration ends. This part belongs to the prior art and will not be elaborated here.

[0030] Next, for the fusion calibration method of the point cloud data and the infrared data disclosed in this embodiment, performance tests are carried out. The test results are as Figure 6 shown in the bar chart in the lower right corner. The average error is 1.5 pixels, and the maximum error is 3.5 pixels, meeting the error requirements of calibration.

[0031] In summary, on the basis of well solving the problem of difficult calibration, the present invention not only improves the calibration work efficiency, but also can improve the accuracy, that is, reduce the error rate of direct acquisition. At the same time, the present invention not only does not need to customize non-standard infrared checkerboards additionally, greatly reducing the cost, but also the position and environment of the color camera used for assisting fusion calibration are not restricted, that is, it does not need to be combined and bound with the synchronous acquisition module (infrared camera + lidar), and only needs to be fixed by any bracket and can normally collect data in the visible light environment, greatly improving the flexibility.

[0032] Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art and related fields without creative efforts shall fall within the protection scope of the present invention.

Claims

1. A method for fusion calibration of point cloud data and infrared data, by rigidly fixing an infrared camera and a laser radar, and collecting infrared data and point cloud data synchronously at different distances and different postures, characterized in that: When the laser radar collects point cloud data, a non-reflective film of the same size as the infrared chessboard is used to cover the infrared chessboard. A color camera with calibrated internal parameters and external parameters of the laser radar is used to synchronously collect color image data at different distances and different postures in the same frame in a visible light environment with the infrared camera. The color image data and the point cloud data collected by the lidar in the same frame are used as input, and the corner point extraction is realized through feature extraction. The color image data is used as the standard, and the infrared image data and point cloud data in the same frame corresponding to the unqualified color image data in the original sample data that cannot extract the corner point are completely eliminated; Use the pre-processed color image data that can extract corner points to correspond to the infrared image data collected in the same frame, and calculate the intrinsic parameter matrix value of the infrared camera; Using the preprocessed infrared image data and point cloud data, the rigid transformation from the lidar to the infrared camera is estimated, and finally the external parameter matrix value is obtained, and the fusion calibration is completed.

2. The method for fusion calibration of point cloud data and infrared data according to claim 1, characterized in that: Based on the calibration results of the color camera and lidar, the collected point cloud is filtered, and the point cloud data of the chessboard itself is retained. Then, based on the filtered point cloud data, the rigid transformation from the lidar to the infrared camera is estimated, and finally the extrinsic parameter matrix value is obtained.

Images