Automatic calibration method of external parameters for non-repetitive scanning 3D lidar and thermal camera

By using calibration plates with spindles in non-repeat scanning 3D lidar and thermal sensing cameras, automatically detecting and identifying features, combined with nonlinear optimization solutions, the problems of low efficiency and unfriendly foreign parameter calibration in the prior art are solved, and the problem of incompetence of foreign parameter calibration in the existing technology is realized. Efficient and accurate automatic calibration of external parameters has been expanded.

CN114782651BActive Publication Date: 2024-11-12CHINA-SINGAPORE INT JOINT RES INST
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210523925.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-14
Publication Date
2024-11-12
Estimated Expiration
2042-05-14

AI Technical Summary

Technical Problem

The prior art is difficult to realize automatic calibration of external parameters between non-repeat scanning 3D lidar and thermal camera, and the existing methods require manual intervention and expertise, which limits their application scenarios and efficiency.

Method used

By using a calibration plate with a spindle, a non-repeat scan 3D lidar and thermal camera is used to acquire three-dimensional point clouds and thermal images, automatically detect and identify the characteristics of the calibration plate, and combine nonlinear optimization solutions to calculate the accurate external parameters.

Benefits of technology

It realizes user-friendly automatic calibration of foreign parameters, improves calibration efficiency and accuracy, is suitable for traditional lidar and other plane calibration objects, simplifies the operation process, is highly adaptable, and is suitable for batch calibration tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114782651B_ABST
    Figure CN114782651B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for automatic calibration of external parameters of a non-repetitive scanning 3D laser radar and a thermal camera, comprising the following steps: heating or cooling a calibration plate, then aligning the non-repetitive scanning 3D laser radar and the thermal camera with the calibration plate, and moving the calibration plate to different positions to shoot the calibration plate, obtaining and storing corresponding three-dimensional point clouds and thermal images; automatically detecting and identifying the calibration plate for the three-dimensional point cloud and extracting features to obtain corresponding 3D coordinate information; automatically detecting and identifying the calibration plate for the thermal image and extracting features to obtain corresponding 2D coordinate information and 3D coordinate information; using corresponding calibration plate feature point pairs detected in the three-dimensional point cloud and the thermal image respectively, solving external parameters through nonlinear optimization, minimizing feature matching errors; and calculating accurate external parameters of the non-repetitive scanning 3D laser radar and the thermal camera; the present invention is easy to use, and ordinary users can accurately perform automatic calibration of external parameters.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of robot perception technology, and in particular to an automatic calibration method for external parameters of a non-repetitive scanning 3D laser radar and a thermal camera. Background Art

[0002] With the advent of the intelligent era, especially the development of mobile robots, the perception applications of non-repetitive scanning 3D LiDAR and thermal cameras are receiving more and more attention. This is mainly due to the low cost of non-repetitive scanning 3D LiDAR and the improvement of the robot's perception ability in low-light environments after combining with thermal cameras. However, the non-repetitive scanning feature also brings new problems to the external parameter calibration between the two: most calibration methods for traditional LiDAR are based on extracting edges for feature extraction based on depth discontinuity, but are not suitable for non-repetitive scanning 3D LiDAR without fixed scanning lines; some recent methods use the reflection intensity difference of LiDAR to detect calibration objects, but they have limitations and cannot be used for calibration objects without obvious intensity characteristics; most calibration methods with target objects require human intervention to locate the calibration objects in the point cloud. This laborious and inefficient method cannot meet the needs of batch calibration tasks.

[0003] In order to fully utilize the complementary information of different sensors, accurate extrinsic and external parameter calibration is indispensable. Currently, there are few calibration technologies that can achieve extrinsic parameter calibration between non-repetitive scanning 3D lidar and thermal cameras. What's worse, the existing calibration methods are not user-friendly and require users to have relevant professional knowledge, which will limit the use scenarios of non-repetitive scanning 3D lidar and thermal cameras. Summary of the invention

[0004] The purpose of the present invention is to overcome the above-mentioned defects in the prior art and provide a method for automatic calibration of external parameters of non-repetitive scanning 3D laser radar and thermal camera; the method is easy to use, user-friendly, does not require too much professional knowledge, and ordinary users can accurately perform automatic calibration of external parameters.

[0005] To achieve the above object, the present invention provides a method for automatic calibration of external parameters of a non-repetitive scanning 3D laser radar and a thermal camera, comprising the following steps:

[0006] Step S1: pre-processing the calibration plate, including heating or cooling the calibration plate, and then aligning the non-repetitive scanning 3D laser radar and thermal camera to the calibration plate;

[0007] Step S2: Collect data; move the non-repetitive scanning 3D laser radar and thermal camera to different positions, shoot the calibration plate through the non-repetitive scanning 3D laser radar and thermal camera, and obtain and store the corresponding three-dimensional point cloud and thermal image;

[0008] Step S3: Automatically detect and identify the calibration plate on the three-dimensional point cloud and extract features to obtain corresponding 3D coordinate information;

[0009] Step S4: Automatically detect and identify the calibration plate on the thermal image and extract features to obtain corresponding 2D coordinate information and 3D coordinate information;

[0010] Step S5: Using the corresponding calibration plate feature point pairs detected in the three-dimensional point cloud and thermal image, respectively, the external parameters are solved through nonlinear optimization to minimize the feature matching error; and accurate non-repetitive scanning 3D laser radar and thermal camera external parameters are calculated.

[0011] Preferably, the shape of the calibration plate in step S1 has a main axis.

[0012] Preferably, a bracket is installed at the lower part of the calibration plate, and a plurality of through holes arranged horizontally and vertically are provided on the calibration plate; a heating device for heating the calibration plate is laid on the back of the calibration plate, the heating device is electrically connected to a temperature controller, and the temperature controller is electrically connected to a power supply.

[0013] Preferably, the calibration plate is made of acrylic resin material.

[0014] Preferably, the step S3 further includes the following steps:

[0015] Step S31: pre-processing the 3D point cloud and performing plane segmentation on the 3D point cloud;

[0016] Step S32: performing intensity value filtering on the segmented plane;

[0017] Step S33: importing the three-dimensional point cloud of the calibration plate template contour, and finding the principal axes of the three-dimensional point clouds on the calibration plate and the template contour respectively;

[0018] Step S34: extracting the contour of the calibration plate in the three-dimensional point cloud;

[0019] Step S35: using the main axis to roughly match the three-dimensional point cloud of the calibration plate and the template contour;

[0020] Step S36: precisely aligning the three-dimensional point cloud of the calibration plate contour and the calibration plate template contour;

[0021] Step S37: Calculate the error of the registration result, compare it with the preset error threshold, and determine whether it is a calibration plate 3D point cloud. If so, output the calibration plate 3D point cloud; if not, output the non-calibration plate 3D point cloud;

[0022] Step S38: extract corresponding point, line and surface features from the output three-dimensional point cloud of the calibration plate, and obtain corresponding 3D coordinate information.

[0023] Preferably, the step S4 also includes dedistorting the thermal image, extracting corresponding point, line and surface features, and obtaining corresponding 2D coordinate information and 3D coordinate information.

[0024] Preferably, step S5 also includes external parameter calibration, pairing the obtained 3D coordinates, solving the external parameters through nonlinear optimization, and obtaining the minimum 3D matching error; pairing the obtained 2D coordinates, solving the external parameters through nonlinear optimization, and obtaining the minimum 2D reprojection error; and finally outputting accurate non-repetitive scanning 3D lidar and thermal camera external parameters.

[0025] The present invention also provides an intelligent robot, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method for automatic calibration of external parameters of a non-repetitive scanning 3D laser radar and a thermal camera as described above.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The calibration method of the present invention has high efficiency and simple process, and only requires one calibration plate. At the same time, the calibration plate is automatically detected and identified from the obtained 3D point cloud without manual intervention; it has strong adaptability and can be applied to traditional laser radars and other planar calibration objects; it has high precision and compares two general nonlinear optimization methods by calibrating multiple calibration plates with different postures in order to obtain the optimal parameters; it is user-friendly and does not require too much professional knowledge. Ordinary users only need to use the non-repetitive scanning 3D laser radar and thermal camera to collect calibration plate data and import the 3D point cloud of the corresponding calibration plate template contour. The algorithm will automatically select reliable data and complete the calibration, and the operation is simple and easy to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 It is a flowchart of the automatic calibration method of external parameters of the non-repetitive scanning 3D laser radar and thermal camera provided by the present invention;

[0030] Figure 2 It is a structural schematic diagram of the calibration plate provided by the present invention;

[0031] Figure 3It is a flow chart of data processing of the obtained three-dimensional point cloud and thermal image provided by the present invention;

[0032] Figure 4 It is a flow chart of data error comparison of the method provided by the present invention.

[0033] Included in the figure are:

[0034] 1. Calibration plate; 12. Bracket; 15. Through hole; 13. Heating device; 14. Temperature controller. DETAILED DESCRIPTION

[0035] The technical scheme in this embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in this embodiment of the present invention. Obviously, the described embodiment is one embodiment of the present invention, not all embodiments of the present invention. Based on this embodiment of the present invention, all other embodiments of the present invention obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Embodiment 1

[0037] Please refer to Figures 1 to 4 Embodiment 1 of the present invention provides a method for automatic calibration of external parameters of a non-repetitive scanning 3D laser radar and a thermal camera.

[0038] First, the method provided by the present invention is generally described: the sensor kit of the present invention includes a non-repetitive scanning 3D laser radar and a thermal camera; the calibration plate 1 used is as follows: Figure 2 As shown; this method can be extended to any calibration plate 1, as long as the shape of the calibration plate 1 used has a principal axis.

[0039] Furthermore, in this embodiment, the calibration plate 1 is made of acrylic resin, a bracket 12 is installed at the lower part of the calibration plate 1, and a plurality of through holes 15 arranged horizontally and vertically are provided on the calibration plate 1; Figure 2 As shown, there are four through holes 15, two in the horizontal direction and two in the vertical direction. A heating device 13 for heating the calibration plate 1 is provided on the back of the calibration plate 1. The heating device 13 is electrically connected to a temperature controller 14, and the temperature controller 14 is electrically connected to a power supply.

[0040] Furthermore, in this embodiment, the thermostat 14 is connected to a power source and started, and the thermostat 14 controls the heating device 13 to heat the calibration plate 1 to a target temperature.

[0041] Next, a method for automatic calibration of external parameters of a non-repetitive scanning 3D laser radar and thermal camera is introduced in detail. Figure 1 Shown is a flow chart of the present invention.

[0042] Reference Figure 1 The present invention mainly includes the following steps: Step S1: pretreatment of the calibration plate; including heating or cooling the calibration plate, and then aligning the non-repetitive scanning 3D laser radar and the thermal camera with the calibration plate.

[0043] Step S2: Collect data; move the non-repetitive scanning 3D laser radar and thermal camera to different positions, and shoot the calibration plate by the non-repetitive scanning 3D laser radar and thermal camera. The shooting can last for a period of time, so as to obtain and store the corresponding three-dimensional point cloud and thermal image.

[0044] Step S3: Automatically detect and identify the calibration plate on the three-dimensional point cloud and extract features to obtain corresponding 3D coordinate information.

[0045] Further, such as Figure 3 As shown, the step S3 also includes the following steps:

[0046] Step S31: pre-processing the collected 3D point cloud and performing plane segmentation on the 3D point cloud;

[0047] Step S32: performing intensity value filtering on the segmented plane;

[0048] Step S33: importing the three-dimensional point cloud of the calibration plate template contour, and finding the principal axes of the three-dimensional point clouds on the calibration plate 1 and the template contour respectively;

[0049] Step S34: extracting the contour of the calibration plate 1 in the three-dimensional point cloud;

[0050] Step S35: using the main axis to roughly match the three-dimensional point cloud of the calibration plate 1 and the template contour;

[0051] Step S36: precisely aligning the three-dimensional point cloud of the calibration plate 1 contour and the calibration plate template contour;

[0052] Step S37: Figure 4 As shown, the error of the registration result is calculated and compared with the preset error threshold to determine whether it is a calibration plate 3D point cloud. If so, the calibration plate 3D point cloud is output; if not, the non-calibration plate 3D point cloud is output;

[0053] Step S38: extract corresponding point, line and surface features from the output three-dimensional point cloud of the calibration plate, and obtain corresponding 3D coordinate information.

[0054] Step S4: Automatically detect and identify the calibration plate and extract features from the thermal image to obtain corresponding 2D coordinate information and 3D coordinate information; Figure 3As shown, the step S4 also includes dedistorting the thermal image, extracting corresponding point, line and surface features, and obtaining corresponding 2D coordinate information and 3D coordinate information.

[0055] Step S5: Using the corresponding calibration plate feature point pairs detected in the three-dimensional point cloud and the thermal image, the extrinsic parameters are solved by nonlinear optimization to minimize the feature matching error; the accurate non-repetitive scanning 3D laser radar and thermal camera extrinsic parameters are calculated; Figure 3 As shown, step S5 also includes external parameter calibration, pairing the obtained 3D coordinates, solving the external parameters through nonlinear optimization, and obtaining the minimum 3D matching error; pairing the obtained 2D coordinates, solving the external parameters through nonlinear optimization, and obtaining the minimum 2D reprojection error; finally, outputting accurate non-repetitive scanning 3D lidar and thermal camera external parameters.

[0056] This method has low complexity. To achieve the calibration of the sensor suite, the existing technology often requires the use of multiple different calibration plates, which is a cumbersome process. However, the external parameter calibration method of the sensor suite of the present invention only requires one calibration plate to complete the external parameter calibration. This improves the cumbersome and user-unfriendly calibration process of the existing non-repetitive scanning 3D laser radar and thermal camera, thereby expanding the application field of non-repetitive scanning 3D laser radar and thermal camera.

[0057] Embodiment 2

[0058] Embodiment 2 of the present invention provides an intelligent robot, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a method for automatic calibration of external parameters of a non-repetitive scanning 3D laser radar and a thermal camera as described in Embodiment 1.

[0059] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for automatic calibration of external parameters of non-repetitive scanning 3D laser radar and thermal camera, characterized by: The following steps are involved: Step S1: pre-processing the calibration plate, including heating or cooling the calibration plate, and then aligning the non-repetitive scanning 3D laser radar and the thermal camera with the calibration plate; the shape of the calibration plate (1) has a main axis; Step S2: Collect data; move the non-repetitive scanning 3D laser radar and thermal camera to different positions, shoot the calibration plate through the non-repetitive scanning 3D laser radar and thermal camera, and obtain and store the corresponding three-dimensional point cloud and thermal image; Step S3: Automatically detect and identify the calibration plate on the three-dimensional point cloud and extract features to obtain corresponding 3D coordinate information; The step S3 also includes the following steps: Step S31: pre-processing the 3D point cloud and performing plane segmentation on the 3D point cloud; Step S32: performing intensity value filtering on the segmented plane; Step S33: importing the three-dimensional point cloud of the calibration plate template contour, and finding the principal axes of the three-dimensional point clouds on the calibration plate (1) and the template contour respectively; Step S34: extracting the contour of the calibration plate (1) in the three-dimensional point cloud; Step S35: using the main axis to roughly match the three-dimensional point cloud of the calibration plate (1) and the template contour; Step S36: precisely aligning the three-dimensional point cloud of the calibration plate (1) contour with the calibration plate template contour; Step S37: Calculate the error of the registration result, compare it with the preset error threshold, and determine whether it is a calibration plate 3D point cloud. If so, output the calibration plate 3D point cloud; if not, output the non-calibration plate 3D point cloud; Step S38: extracting corresponding point, line and surface features from the output three-dimensional point cloud of the calibration plate, and obtaining corresponding 3D coordinate information; Step S4: Automatically detect and identify the calibration plate on the thermal image and extract features to obtain corresponding 2D coordinate information and 3D coordinate information; Step S5: Using the corresponding calibration plate feature point pairs detected in the three-dimensional point cloud and thermal image, respectively, the external parameters are solved through nonlinear optimization to minimize the feature matching error; and accurate non-repetitive scanning 3D laser radar and thermal camera external parameters are calculated.

2. The method for automatic calibration of external parameters of a non-repetitive scanning 3D laser radar and a thermal camera according to claim 1, characterized in that: A bracket (12) is installed at the bottom of the calibration plate (1), and a plurality of through holes (15) are arranged horizontally and vertically on the calibration plate (1); a heating device (13) for heating the calibration plate (1) is provided on the back of the calibration plate (1), the heating device (13) is electrically connected to a temperature controller (14), and the temperature controller (14) is electrically connected to a power supply.

3. The method for automatic calibration of external parameters of non-repetitive scanning 3D laser radar and thermal camera according to claim 2, characterized in that: The calibration plate (1) is a component made of acrylic resin material.

4. The method for automatic calibration of external parameters of a non-repetitive scanning 3D laser radar and a thermal camera according to claim 1, characterized in that: The step S4 also includes dedistorting the thermal image, extracting corresponding point, line and surface features, and obtaining corresponding 2D coordinate information and 3D coordinate information.

5. The method for automatic calibration of external parameters of a non-repetitive scanning 3D laser radar and a thermal camera according to claim 4, characterized in that: The step S5 also includes external parameter calibration, pairing the obtained 3D coordinates, solving the external parameters through nonlinear optimization, and obtaining the minimum 3D matching error; pairing the obtained 2D coordinates, solving the external parameters through nonlinear optimization, and obtaining the minimum 2D reprojection error; and finally outputting accurate non-repetitive scanning 3D lidar and thermal camera external parameters.

6. An intelligent robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements a method for automatic calibration of external parameters of a non-repetitive scanning 3D laser radar and a thermal camera as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Laser radar and camera combined calibration method

    CN112669393A

  • Simultaneous internal and external parameter calibration method for multi-source heterogeneous sensor

    CN113327289A

  • Calibration method and calibration apparatus for external parameters of multiple groups of laser radars, and computer storage medium

    WO2021253193A1