A device and method for joint calibration of a visible light camera and an infrared camera

Abstract

The present invention discloses a visible light camera, an infrared camera joint calibration device and a calibration method, which adopt a heating calibration plate composed of a black hollow heating plate, an LED cylindrical lamp post, a housing, a power supply and a switch; wherein, the hollow heating plate is arranged in the housing and connected to the power supply through the housing, and a plurality of through holes are uniformly arranged therein, and the corresponding LED cylindrical lamp posts are arranged in the through holes, and the LED cylindrical lamp posts are connected to the power supply through the housing, and the switch is used to control the power supply output to make the hollow heating plate heat or the LED cylindrical lamp posts emit light; when the hollow heating plate heats, a uniform temperature difference is formed between the through holes for the infrared camera to capture a clear temperature image; when the LED cylindrical lamp posts emit light, a color difference is formed with the black hollow heating plate for the visible light camera to capture a clear image. By adopting the technical scheme of the present invention, it can be used for the calibration of both the visible light camera and the infrared camera at the same time.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine vision camera calibration, and particularly relates to a device and a calibration method for the joint calibration of a visible light camera and an infrared camera. Background Art

[0002] A visible light camera can obtain rich high-resolution and high-definition texture detail information, and an infrared camera is sensitive to the temperature change of objects in a scene and can obtain the temperature thermodynamic distribution information of the objects. The limitations of the two technologies are independent of each other and usually do not occur simultaneously. Therefore, the joint calibration of a visible light camera and an infrared camera for imaging and detection has been widely applied in industries such as remote sensing, security monitoring, and industrial inspection.

[0003] At present, most of the devices and methods for dual-camera calibration are based on binocular visible light cameras. The monocular and binocular calibration devices and methods based on visible light cameras have been relatively mature. The calibration device for a visible light camera often uses a checkerboard calibration plate, and the calibration method proposed by Zhang Zhengyou is used to estimate the parameters of the camera. There is less research on the calibration algorithms for dual cameras under different spectra, and there are the following problems in the calibration of camera systems involving infrared cameras:

[0004] The standard checkerboard method for camera calibration depends on the color difference of the checkerboard. However, due to the characteristics of the infrared camera, only the temperature radiation difference points can be obtained, and the checkerboard calibration plate for the visible light camera cannot form a thermal radiation difference. For the design of the infrared camera calibration plate, a heatable checkerboard is generally used. By constructing checkerboards of two different materials, a large thermal radiation rate and temperature difference are created, so that the infrared camera can obtain a thermogram presenting a checkerboard pattern. However, the thermal crosstalk between adjacent checkerboards is very serious, so the temperature gradient between adjacent checkerboards is very small, which will lead to a decrease in the corner detection accuracy. In addition, since the visible light camera and the infrared camera image under different spectra, a calibration plate that can act on both the visible light camera and the thermodynamic distribution needs to be proposed.

[0005] Due to the small resolution and large noise of the infrared camera, the images it captures are often not as clear as those captured by the visible light camera, and the brightness contrast of the boundary area with a large temperature difference is not as clear as in the actual situation, which makes the calibration of the infrared camera very difficult. Summary of the Invention

[0006] To solve the above problems, the present invention provides a device and a method for the joint calibration of a visible light camera and an infrared camera, which uses a heat-generating and hollowed calibration plate that can be used for the calibration of both the visible light camera and the infrared camera at the same time.

[0007] In order to overcome the defects of the prior art, the specific technical solution of the present invention is as follows:

[0008] A combined calibration device for a visible light camera and an infrared camera, the calibration device adopts a heating calibration plate, and is composed of a black hollow heating plate, an LED cylindrical lamp post, a shell, a power supply, and a switch; wherein the hollow heating plate is arranged in the shell and connected to the power supply through the shell, a plurality of through holes are evenly arranged therein, and corresponding LED cylindrical lamp posts are arranged in the through holes, and the LED cylindrical lamp posts are connected to the power supply through the shell, and the switch is used to control the power supply output to make the hollow heating plate heat up or the LED cylindrical lamp post emit light;

[0009] When the hollow heating plate generates heat, a uniform temperature difference is formed between the hollow heating plate and the through hole, so that the infrared camera can capture a clear temperature image.

[0010] When the LED cylindrical lamp post emits light, a color difference is formed with the black hollow heating plate, so that a visible light camera can capture a clear image.

[0011] Specifically, the hollow heating plate is a conductive high-temperature resistance heating plate, which is connected to the power supply through the shell and can be heated evenly when the power is turned on. The plate is provided with 9×12 uniform circular through holes. When the power is turned on, the plate can be heated by adjusting the switch provided on the shell. The hollow circular through hole part does not heat up, and the rest of the part heats up evenly, so that a uniform temperature difference is formed between the plate and the circular through hole, so that the infrared camera can capture a clear temperature image.

[0012] Specifically, the number, specifications and dimensions of the LED cylindrical lamp posts are the same as those of the hollow part of the hollow heating plate. They are embedded in the hollow part of the hollow heating plate. A layer of insulation material is applied on the contact surface between the LED cylindrical lamp post and the hollow part of the hollow heating plate to prevent thermal crosstalk when the hollow heating plate is heated. The LED cylindrical lamp post is connected to the power supply through the shell. After the power is turned on, it can be lit by adjusting the switch set on the shell. After lighting, it emits white light, forming a strong color difference with the black hollow heating plate, allowing the visible light camera to capture clear images.

[0013] Specifically, the shell is made of insulating material, wrapping the hollow heating plate and the LED cylindrical lamp column to prevent the heat from being conducted elsewhere when the hollow heating plate is heated. The shell has built-in wires connected to the power supply to control the heating of the hollow heating plate and the lighting of the LED cylindrical lamp column through a switch.

[0014] The present invention also discloses a method for joint calibration of a visible light camera and an infrared camera, comprising the following steps:

[0015] Step S1: The visible light camera and the infrared camera respectively acquire the calibration plate image;

[0016] Step S2: performing image processing on the acquired calibration plate image and extracting feature points;

[0017] Step S3: The visible light camera and the infrared camera are respectively calibrated individually;

[0018] Step S4: On the basis of the individual camera calibration in Step S3, the visible light camera and the infrared camera are jointly calibrated;

[0019] Among them, in Step S1, a heating calibration plate is adopted. The heating calibration plate is composed of a black hollow heating plate, an LED cylindrical lamp post, a housing, a power supply, and a switch; when the calibration plate is powered on, the hollow heating plate is evenly heated through the switch, and the LED cylindrical lamp post is lit; under the visible light camera, the hollow heating plate appears black, and after the LED cylindrical lamp post is lit, it emits uniform white light, forming a color difference, and the visible light camera can clearly capture the calibration plate image; under the infrared camera, the hollow heating plate is evenly heated, the plate generates high temperature, and after the LED cylindrical lamp post is lit, the temperature characteristics change little, forming a temperature difference, and the infrared camera can clearly capture the thermal map of the calibration plate.

[0020] As a further improvement scheme, the method for jointly calibrating the visible light camera and the infrared camera is as follows:

[0021] Step 1: Build an experimental platform. Fix the visible light camera and the infrared camera with brackets, and place the calibration plate at a distance waiting to be photographed.

[0022] Step 2: Collect images. When collecting images, the calibration plate is powered on, the hollow heating plate is evenly heated through the switch, and the LED cylindrical lamp post is lit. Under the visible light camera, the hollow heating plate appears black, and after the LED cylindrical lamp post is lit, it emits uniform white light, forming a color difference, and the visible light camera can clearly capture the calibration plate image; under the infrared camera, the hollow heating plate is evenly heated, the plate generates high temperature, and after the LED cylindrical lamp post is lit, the temperature characteristics change little, forming a temperature difference, and the infrared camera can clearly capture the thermal map of the calibration plate. When shooting, ensure that the imaging size of the calibration plate in each image approximately accounts for one-fourth of the entire image. Divide the field of view of the camera into four quadrants. When shooting, ensure that the position of the calibration plate is evenly distributed in each quadrant, and the calibration plates in each quadrant have different directions and angles of inclination. All the calibration plate images combined should be able to cover the measurement space and the field of view under the entire field of view in the actual working condition. The number of calibration images is recommended to be 15 - 25.

[0023] Step 3: Image processing. The image processing process includes image denoising and tone mapping for the visible light image and the infrared image. Since the visible light image can capture higher-resolution and high-clarity texture detail information, the tone mapping step can be selected not to be performed.

[0024] Specifically, Gaussian filtering can be used to perform denoising processing on the images.

[0025] Specifically, tone mapping is performed on the denoised image to enhance the image contrast. Compared with the calibration pattern in the visible light image, the intensity difference between the bright and dark areas in the infrared image is much smaller than that in the visible light image. To obtain a clearer circular through-hole and calibration plate boundary in the infrared image, a tone mapping algorithm is designed to optimize the pixel brightness intensity values in the infrared image, enabling each pixel brightness intensity value to be adaptively mapped to white and black, thereby enhancing the contrast of the infrared image.

[0026] Specifically, the tone mapping processing function f(I (i,j) ) is given by the following formula:

[0027]

[0028] where

[0029] is the intensity of the pixel at the i-th row and j-th column in the mapped infrared image;

[0030] I (i,j) is the intensity of the pixel at the i-th row and j-th column in the infrared image;

[0031] is the average intensity of the brightest point closest to and the darkest point closest to the pixel at the i-th row and j-th column;

[0032] is calculated from Equation ;

[0033] η is the proportionality coefficient.

[0034] Step Four: Feature point extraction. This process is divided into two steps:

[0035] (1) Extract the contour of the circular through-hole of the calibration plate in the image. First, use the Canny edge detection operator to detect the edge contour of the circular through-hole in the image. Then, use the least squares method to fit the extracted edge contour into an ellipse. Due to lens distortion and perspective transformation, the projection of the circular through-hole is generally not circular but approximately elliptical.

[0036] (2) Based on the ellipse fitted above, extract the midpoints of each contour and establish the midpoint coordinates. In the image coordinate system, take the pixel point in the upper left corner as the origin, and based on this, establish a coordinate system to define the coordinates of the midpoint of the detected circular through-hole.

[0037] Specifically, the method for finding the midpoint is as follows: Detect the upper, lower, left, and right boundary points of the edge contour, and connect the upper and lower boundary points and the left and right boundary points respectively. The intersection point of the line segments is the midpoint at that location.

[0038] Step 5: Calibration of the visible light camera and the infrared camera individually. An ideal camera can be regarded as a pinhole imaging model. The relationship between the 3D spatial points in the three-dimensional real world and the projection coordinates of the points in the image coordinate system can be described by the following formula.

[0039]

[0040] Among them,

[0041] is the homogeneous coordinate of the 2D pixel point m = [u, v] in the image coordinate system; T of;

[0042] is the homogeneous coordinate of the 3D spatial point M = [X w , Y w , Z w in the world coordinate system; T of;

[0043] is the internal parameter matrix of the camera,, (f x , f y ) is the focal length in pixels, (u0, v0) is the principal point of the camera, in pixels;

[0044] [R T] is the external parameter matrix of the camera, R is a 3×3 rotation matrix, and T is a 3×1 translation matrix.

[0045] The above internal and external parameter matrices of the camera do not consider lens distortion because the ideal pinhole model does not contain a lens. The lens of a real camera will inevitably produce distortion, which can be corrected by the following equation:

[0046]

[0047] Among them,

[0048] (u, v) is the pixel coordinate of the ideal (i.e., undistorted) image coordinate system;

[0049] (u′, v′) is the pixel coordinate of the actual image coordinate system;

[0050] k1, k2, k2 are the radial distortion coefficients of the camera;

[0051] p1, p2 are the tangential distortion coefficients of the camera;

[0052] r 2 = u 2 + v 2 .

[0053] Step 6: Joint calibration of visible light camera and infrared camera. Based on the parameter matrices of the visible light camera and the infrared camera obtained from the single camera calibration in Step 5, establish the positions between the two cameras in the world coordinate system:

[0054]

[0055] Among them,

[0056] [X vis ,Y vis ,Z vis T and [X ir ,Y ir ,Z ir T are the coordinates of the visible light camera and the infrared camera in the world coordinate system respectively;

[0057] R and T are the rotation matrix and translation matrix of the binocular system respectively.

[0058] The beneficial effects of the present invention are:

[0059] (1) A calibration plate with heat-generating hollowing proposed by the present invention. The calibration plate itself uses high-temperature resistance materials, which can effectively prevent the problem of uneven heating during heating; the calibration plate design with 9×12 evenly distributed circles hollowed out is adopted, and the center coordinates of the circles are used to replace the corner coordinates between the checkerboards of the traditional checkerboard calibration plate, which can effectively reduce the influence of thermal crosstalk phenomenon under the shooting of the infrared camera and has high robustness; in addition, this calibration plate can be used for the calibration of both visible light cameras and infrared cameras at the same time.

[0060] (2) Aiming at the problems that the infrared camera has low resolution and large noise, and the images it captures are often not as clear as those captured by the visible light camera, and the brightness contrast of the boundary area with large temperature difference is not as clear as in the actual situation, a tone mapping method proposed by the present invention can optimize the brightness intensity values of the pixels in the infrared image, enhance the contrast of the infrared image, and is beneficial to obtaining clearer circular through-holes and calibration plate boundaries in the infrared image.

[0061] (3) Due to the lens distortion and perspective transformation of the camera lens, the projection of the circular through-hole is generally not circular but approximately elliptical, and finding the center of the ellipse is more complicated than finding the center of the circle. A method for finding the center of the ellipse proposed by the present invention is convenient for more easily obtaining the feature points required for camera calibration. Description of the Drawings

[0062] Figure 1 is the front view of the calibration plate described in the embodiment of the present invention;

[0063] Figure 2 is the structural schematic diagram of the calibration plate described in the embodiment of the present invention; ​​

[0064] Figure 3 Schematic diagram of camera calibration according to an embodiment of the present invention;

[0065] Figure 4 Flow chart of infrared camera calibration according to an embodiment of the present invention;

[0066] Figure 5 Schematic diagram of dividing quadrants of the field of view of an infrared camera according to an embodiment of the present invention;

[0067] Figure 6 Schematic diagram of finding the midpoint according to an embodiment of the present invention;

[0068] Figure 7 Flow chart of joint calibration of visible light camera and infrared camera according to an embodiment of the present invention. Detailed implementation manners

[0069] The present invention will be further described below in conjunction with specific embodiments.

[0070] Embodiment 1

[0071] As Figure 1 — Figure 2 shown, Figure 1 is the front view of a calibration device 1 designed by the present invention, and its structure is as Figure 2 shown. The calibration device 1 adopts a heating calibration plate, which is composed of a hollowed-out heating plate 1-1, an LED cylindrical lamp post 1-2, a housing 1-3, a power supply 1-4, and a switch 1-5.

[0072] Among them, the hollowed-out heating plate 1-1 is arranged in the housing 1-3 and connected to the power supply 1-4 through the housing 1-3. A plurality of through holes are evenly arranged therein, and an adapted LED cylindrical lamp post 1-2 is arranged in the through holes. The LED cylindrical lamp post 1-2 is connected to the power supply 1-4 through the housing 1-3. The switch is used to control the output of the power supply 1-4 to make the hollowed-out heating plate 1-1 heat up or the LED cylindrical lamp post 1-2 emit light;

[0073] When the hollowed-out heating plate 1-1 heats up, a uniform temperature difference is formed between it and the through holes, so as to be used for the infrared camera to capture clear temperature images;

[0074] When the LED cylindrical lamp post 1-2 emits light, a color difference is formed with the black hollowed-out heating plate, so as to be used for the visible light camera to capture clear images.

[0075] The hollow heating plate 1-1 is a conductive high-temperature resistance heating plate. This plate is connected to the power supply 1-4 through the housing 1-3 (the wiring part is not shown). When the power supply 1-4 is turned on, it can be evenly heated, and a uniform circular through-hole of 9×12 is set on this plate. When the power supply 1-4 is turned on, the plate can generate heat by adjusting the switch 1-5 set on the housing 1-3. Among them, the hollow circular through-hole part does not generate heat, and the rest part generates heat evenly, so that a uniform temperature difference is formed between the plate and the circular through-hole, enabling the infrared camera to capture clear temperature images.

[0076] The quantity specification size of the LED cylindrical lamp post 1-2 is the same as that of the hollow part of the hollow heating plate 1-1, and it is embedded in the hollow part of the hollow heating plate 1-1. A layer of heat insulation material (not shown) is smeared on the contact surface between the LED cylindrical lamp post 1-2 and the hollow part of the hollow heating plate 1-1 to prevent heat crosstalk when the hollow heating plate 1-1 is heated. The LED cylindrical lamp post 1-2 is connected to the power supply 1-4 through the housing 1-3 (the wiring part is not shown). After the power supply 1-4 is turned on, it can be lit by adjusting the switch 1-5 set on the housing 1-3. After being lit, it emits white light, forming a strong color difference with the black hollow heating plate, enabling the visible light camera to capture clear images.

[0077] The housing 1-3 is made of heat insulation material, wrapping the hollow heating plate 1-1 and the LED cylindrical lamp post 1-2. The purpose is to prevent the heat from the hollow heating plate 1-1 from being conducted elsewhere when it generates heat, and it internally contains wires connected to the power supply 1-4 to control whether the hollow heating plate 1-1 generates heat and the LED cylindrical lamp post 1-2 is lit through the switch 1-3.

[0078] As Figure 3 shown, when the camera is calibrated, the visible light camera 2 and the infrared camera 3 are fixed by the bracket 4, and the calibration board 1 is movable.

[0079] Refer to Figure 4 , the present invention also provides a calibration method for single-camera calibration of an infrared camera. The single-camera calibration of the infrared camera 3 is carried out by using the above calibration device, including the following steps:

[0080] S101, obtain images of the calibration board 1 at different positions. Under the infrared camera 3, the hollow heating plate 1-1 generates heat evenly, this plate generates high temperature, and the temperature characteristic change of the LED cylindrical lamp post 1-2 is relatively small, forming a temperature difference. The infrared camera 3 can clearly capture the thermal image of the calibration board. When shooting, ensure that the imaging size of the calibration board in each image approximately accounts for one-fourth of the entire image, as Figure 5As shown, the field of view of the camera is divided into four quadrants. During shooting, ensure that the positions of the calibration plates are evenly distributed in each quadrant, and the calibration plates in each quadrant have different directions and angles of inclination. All the calibration plate images combined should be able to cover the measurement space and the field of view within the entire field of view under actual working conditions. The number of calibration images is recommended to be between 15 and 25.

[0081] S102, perform noise reduction processing on the image using Gaussian filtering.

[0082] S103, perform tone mapping on the image to enhance the contrast. Compared with the calibration pattern in the visible light image, the intensity difference between the bright and dark areas in the infrared image is much smaller than that in the visible light image. To obtain a clearer boundary between the circular through-hole and the calibration plate in the infrared image, a tone mapping algorithm is designed to optimize the pixel light and dark intensity values in the infrared image, enabling each pixel light and dark intensity value to be adaptively mapped into white and black, thereby enhancing the contrast of the infrared image.

[0083] Specifically, the tone mapping processing function f(I (i,j) ) is given by the following formula:

[0084]

[0085] where,

[0086] is the intensity of the pixel at the i-th row and j-th column in the mapped infrared image;

[0087] I (i,j) is the intensity of the pixel at the i-th row and j-th column in the infrared image;

[0088] is the average intensity of the brightest point closest to and the darkest point closest to the pixel at the i-th row and j-th column;

[0089] is calculated from the formula ;

[0090] η is the proportionality coefficient.

[0091] S104, use the Canny edge detection operator to detect the edge contour of the circular through-hole in the image. Due to lens distortion and perspective transformation, the projection of the circular through-hole is generally not circular but approximately elliptical. Use the least squares method to fit the extracted edge contour into an ellipse.

[0092] S105, extract the midpoints of each contour and establish the midpoint coordinates. In the image coordinate system, take the pixel point in the upper left corner as the origin, and based on this, establish a coordinate system to define the coordinates of the midpoint of the detected circular through-hole.

[0093] Specifically, seeFigure 6 , The method for finding the midpoint is as follows: Detect the upper, lower, left, and right boundary points (x u , y u ), (x d , y d ) (x l , y l ) (x r , y r ) of the edge contour. Connect (x u , y u ) (x d , y d ) and (x l , y l ) (x r , y r ). The intersection point of the line segments is the midpoint (x c , y c ) at that location.

[0094] S106, Camera calibration. The 2D pixel point in the image coordinate system is m = [u, v] T , and the 3D spatial point in the world coordinate system is M = [X w , Y w , Z w T , and the corresponding homogeneous coordinates are respectively An ideal camera can be regarded as a pinhole imaging model. The relationship between the 3D spatial point in the three-dimensional real world and the projected coordinates of this point in the image coordinate system can be described by Equation (2).

[0095]

[0096] Among them,

[0097] is the internal parameter matrix of the camera, (f x , f y ) is the focal length in pixels, (u0, v0) is the principal point of the camera, in pixels;

[0098] [R T] is the external parameter matrix of the camera, R is a 3×3 rotation matrix, and T is a 3×1 translation matrix.

[0099] The above camera matrix does not consider lens distortion because the ideal pinhole model does not contain a lens. The lens of a real camera will inevitably produce distortion, which can be corrected by Equation (3).

[0100]

[0101] Among them,

[0102] ​(u, v) are the pixel coordinates of the ideal (i.e., distortion-free) image coordinate system;

[0103] (u′, v′) are the pixel coordinates of the actual image coordinate system;

[0104] k1, k2, k2 are the radial distortion coefficients of the camera;

[0105] p1, p2 are the tangential distortion coefficients of the camera;

[0106] r 2 = u 2 + v 2 .

[0107] Based on the above operations, single-camera calibration of the infrared camera can be achieved.

[0108] Example Two

[0109] The joint calibration process of the visible light camera and the infrared camera is as Figure 7 shown and includes the following steps:

[0110] S1, the visible light camera and the infrared camera acquire calibration board images. Refer to Figure 3 , the visible light camera 2 and the infrared camera 3 are fixed by the bracket 4, and the calibration board 1 is movable. When performing image acquisition, the calibration board 1 is powered on. The hollow heating plate 1-1 is uniformly heated through the switch 1-5, and the LED cylindrical lamp post 1-2 is lit. Under the visible light camera 2, the hollow heating plate 1-1 appears black, and after the LED cylindrical lamp post 1-2 is lit, it emits uniform white light, forming a color difference, and the visible light camera 2 can clearly capture the calibration board image; under the infrared camera 3, the hollow heating plate 1-1 is uniformly heated, and the plate generates high temperature. After the LED cylindrical lamp post 1-2 is lit, the temperature characteristics change little, forming a temperature difference, and the infrared camera 3 can clearly capture the thermal image of the calibration board. When shooting, ensure that the imaging size of the calibration board in each image approximately accounts for one-fourth of the entire image. As Figure 5 shown, divide the field of view of the camera into four quadrants. When shooting, ensure that the position of the calibration board is evenly distributed in each quadrant, and the calibration board in each quadrant has different directions and angles of inclination. All the calibration board images combined should be able to cover the measurement space and field of view under the entire field of view in the actual working condition. The number of calibration images is recommended to be 15 - 25.

[0111] S2, image processing, extract feature points. For the image processing and extraction of feature points for the visible light image and the infrared image, refer to steps S102 - S105 in Example One, Figure 4 Since the visible light image can capture higher-resolution and high-clarity texture detail information, Figure 4 step S103 in

[0112] S3. Calibrate the visible light camera and the infrared camera separately. For the specific method, please refer to Step S106 in Embodiment 1. Figure 4

[0113] S4. Binocular calibration. The following equations (4) and (5) hold for the visible light camera 2 and the infrared camera 3 in the world coordinate system.

[0114]

[0115]

[0116] Among them,

[0117] [X T , X T , Z T T is the midpoint coordinate of a circular through-hole at a certain position on the calibration board 1 in the world coordinate system;

[0118] [X vis , Y vis , Z vis T and [X ir , Y ir , Z ir T are the coordinates of the visible light camera 2 and the infrared camera 3 in the world coordinate system respectively;

[0119] R vis , T vis are the rotation and translation matrices of [X T , Y T , Z T T relative to the visible light camera 2;

[0120] R ir , T ir are the rotation and translation matrices of [X T , Y T , Z T T relative to the infrared camera 3.

[0121] Subtract the two equations to eliminate P w to obtain Equation (6).

[0122]

[0123] That is the rotation matrix R and the translation matrix T of the binocular system.

[0124] Therefore, the position of the infrared camera 3 relative to the visible light camera 2 can be expressed as Equation (7). ​​​​​​

[0125]

[0126] Based on the above operations, the joint calibration of the visible light camera 2 and the infrared camera 3 can be achieved.

[0127] The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

[0128] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visible light camera and an infrared camera joint calibration device, characterized in that The calibration device uses a heating calibration plate, which is composed of a black hollow heating plate, LED cylindrical lamp posts, a housing, a power supply and a switch. Among them, the hollow heating plate is arranged in the housing and connected to the power supply through the housing. A plurality of through holes are evenly arranged in it, and the corresponding LED cylindrical lamp posts are arranged in the through holes. The LED cylindrical lamp posts are connected to the power supply through the housing. The switch is used to control the power output to make the hollow heating plate heat up or the LED cylindrical lamp posts emit light. When the hollow heating plate heats up, a uniform temperature difference is formed between the plate and the through holes, which is used for the infrared camera to capture clear temperature images. When the LED cylindrical lamp posts emit light, a color difference is formed with the black hollow heating plate, which is used for the visible light camera to capture clear images. The hollow heating plate is a conductive high-temperature resistance heating plate. Among them, the hollow through-hole part does not heat up, so that a uniform temperature difference is formed between the plate and the through holes. The LED cylindrical lamp posts are embedded in the hollow part of the hollow heating plate. The number, specification and size are the same as those of the hollow part of the hollow heating plate. After being lit, they emit white light, forming a strong color difference with the black hollow heating plate. A layer of heat-insulating material is applied to the contact surface between the LED cylindrical lamp posts and the hollow part of the hollow heating plate. The housing is made of heat-insulating material and is used to wrap the hollow heating plate and the LED cylindrical lamp posts.

2. A method for joint calibration of a visible light camera and an infrared camera using the device according to claim 1, characterized in that, It includes the following steps: Step S1: The visible light camera and the infrared camera respectively obtain calibration plate images. Step S2: Perform image processing on the obtained calibration plate images and extract feature points. Step S3: The visible light camera and the infrared camera respectively perform single-camera calibration. Step S4: On the basis of the single-camera calibration in Step S3, the visible light camera and the infrared camera are jointly calibrated. In Step S1, a heating calibration plate is used. The heating calibration plate is composed of a black hollow heating plate, LED cylindrical lamp posts, a housing, a power supply and a switch. The calibration plate is connected to the power supply, and the hollow heating plate is evenly heated through the switch, and the LED cylindrical lamp posts are lit. Under the visible light camera, the hollow heating plate appears black, and the LED cylindrical lamp posts emit uniform white light after being lit, forming a color difference. The visible light camera can clearly capture the calibration plate image. Under the infrared camera, the hollow heating plate is evenly heated, generating high temperature, and a uniform temperature difference is formed between the plate and the through holes. The infrared camera can clearly capture the calibration plate thermal map.

3. The method for jointly calibrating a visible light camera and an infrared camera according to claim 2, wherein In Step S2, image denoising and tone mapping are performed on the visible light image and the infrared image. Among them, Gaussian filtering is used to perform noise reduction processing on the image. Tone mapping is performed on the denoised image to enhance the image contrast, and a tone mapping algorithm is designed to optimize the pixel brightness intensity values in the infrared image, so that each pixel brightness intensity value is adaptively mapped into white and black, enabling the enhancement of the infrared image contrast. The tone mapping processing function f(I (i,j) ) is given by the following formula: Among them, is the intensity of the pixel at the i-th row and j-th column in the mapped infrared image; I (i,j) is the intensity of the pixel at the i-th row and j-th column in the infrared image; is the average intensity of the brightest point and the darkest point closest to the pixel at the i-th row and j-th column; Calculated from the formula ; η is the proportionality coefficient.

4. The method for jointly calibrating a visible light camera and an infrared camera according to claim 3, characterized in that, In Step S2, the feature point extraction includes the following steps: (1) Extract the contour of the through holes of the calibration plate in the image. First, use the Canny edge detection operator to detect the edge contour of the through holes in the image. Then, use the least squares method to fit the extracted edge contour into an ellipse. (2) Based on the ellipse fitted above, extract the midpoints of each contour and establish the midpoint coordinates; in the image coordinate system, take the pixel at the upper left corner as the origin, and based on this, establish a coordinate system to define the coordinates of the midpoint of the detected through-hole. Among them, the method for finding the midpoint is: detect the upper, lower, left, and right boundary points of the edge contour, connect the upper and lower boundary points and the left and right boundary points respectively, and the intersection of the line segments is the midpoint at that place.

5. The method for jointly calibrating a visible light camera and an infrared camera according to claim 4, wherein In step S3, regarding the visible light camera and the infrared camera as pinhole imaging models, the relationship between the 3D space points in the three-dimensional real world and the projection coordinates of the points in the image coordinate system can be described by the following formula: Among them, is the homogeneous coordinate of the 2D pixel point m = [u, v] in the image coordinate system T ; The homogeneous coordinates of the 3D spatial point M = [X w , Y w , Z w in the world coordinate system; T ​ is the internal parameter matrix of the camera, where (f x , f y ) is the focal length in pixels, and (u0, v0) is the principal point of the camera in pixels; [RT] is the external parameter matrix of the camera, R is a 3×3 rotation matrix, and T is a 3×1 translation matrix; The above internal and external parameter matrices of the camera do not consider lens distortion. Because the ideal pinhole model does not contain a lens, the lens of a real camera will inevitably produce distortion, which is corrected by the following equation: Among them, (u, v) are the pixel coordinates of the ideal undistorted image coordinate system; (u ′ , v') is the pixel coordinate of the actual image coordinate system; k1, k2, k2 are the radial distortion coefficients of the camera; p1, p2 are the tangential distortion coefficients of the camera; r 2 = u 2 + v 2 .

6. The method for jointly calibrating a visible light camera and an infrared camera according to claim 5, characterized in that In step S4, based on the parameter matrices of the visible light camera and the infrared camera obtained from the single-camera calibration in step S3, establish the positions between the two cameras in the world coordinate system: Among them, [X vis , Y vis , Z vis T 、[X ir , Y ir , Z ir T are the coordinates of the visible light camera and the infrared camera in the world coordinate system, respectively;​​ R and T are the rotation matrix and translation matrix of the binocular system respectively.

Images