A method and system for combined calibration of a thermopile array and a depth camera

Abstract

The application discloses a kind of thermoelectric array and depth camera joint calibration method and system.The application can be simultaneously detected by thermoelectric array and depth camera with spherical calibration object, and the spherical center of the spherical calibration object is used as feature point;The spherical calibration object is set to the position so that the temperature field observed by the thermoelectric array has central symmetry;Change the space pose of the spherical calibration object and synchronously collect the first coordinate set of the spherical center in the thermoelectric array coordinate system and the second coordinate set in the depth camera coordinate system;Based on the first coordinate set and the second coordinate set, the spatial transformation relationship between the thermoelectric array and the depth camera is solved.By the application, the thermoelectric array image coordinates are not dependent on detection, but rely on aiming at a specific location, and use temperature measurement symmetry, aiming accuracy is much higher than detection, so high-precision thermoelectric array image coordinates are obtained.

Description

Technical Field

[0001] This invention relates to the field of calibration technology, specifically a method and system for joint calibration of a thermopile array and a depth camera. Background Technology

[0002] Thermopile arrays can non-contactly detect non-uniform temperature fields and output images. Compared with thermal imagers, they offer advantages such as extremely low cost, small size, and low power consumption, making them a promising alternative for detecting ambient temperature information in many applications. However, thermopile arrays suffer from low resolution, with common pixel specifications of 8×8 or 16×16, while thermal imagers typically have resolutions of 320×240, 640×480, or even higher, severely limiting their applications. Meanwhile, visible light cameras, as a common environmental sensing method, offer advantages such as high resolution and extremely low cost, but they can only detect visible light and cannot detect mid- to long-wavelength radiation, thus limiting their use for temperature monitoring. By fusing data from thermopile arrays and visible light cameras, both temperature measurement and high-resolution imaging can be achieved.

[0003] The foundation for fusing thermopile array images and visible light camera images lies in the joint geometric calibration of the relevant sensor systems, establishing the transformation relationships between the coordinates of the images from each sensor. This presents several technical challenges. First, the thermopile array sensor and the visible light camera operate at different wavelengths; the former is sensitive only to infrared radiation, while the latter is sensitive only to visible light. Second, the positions and orientations of the thermopile array and the visible light camera in the joint system may vary, and their detection directions may even be opposite, which differs significantly from conventional multi-camera systems. Finally, and most importantly, the thermopile array has extremely low resolution, meaning that obtaining high-precision image coordinates of feature points on the calibration object is difficult during calibration, thus hindering the acquisition of high-precision calibration results.

[0004] Multi-camera calibration has many mature technologies, with the Zhang Zhengyou method being a common approach, using a black and white checkerboard pattern for calibration. However, the checkerboard pattern is revealed by reflected visible light and is often indistinguishable in the infrared band. Some improvements have made the checkerboard visible in the infrared band, such as replacing the black and white colors with checkerboard patterns of different emissivity, allowing for differentiation between exposed metal surfaces and painted areas in the infrared. Even so, the infrared contrast is very low, especially for low-resolution thermopile array sensors, where the detection accuracy of checkerboard feature points is extremely low. Furthermore, the checkerboard is a planar calibration object, requiring multiple sensors to be located on the same side of the calibration object; it cannot be used if they are located on opposite sides.

[0005] In fields such as autonomous driving, there is a need and solution for joint calibration of multimodal sensors such as cameras and radar, but the relevant solutions cannot solve the aforementioned problems such as arbitrary angle and low resolution. Summary of the Invention

[0006] The technical problem to be solved by this invention is: how to perform high-precision joint calibration of the thermopile array sensor and the camera in a system composed of a thermopile array and a depth camera, and establish the coordinate system transformation relationship of each sensor.

[0007] The technical solution adopted by this invention to solve the technical problem is as follows:

[0008] In a first aspect, the present invention provides a joint calibration method for a thermopile array and a depth camera, comprising:

[0009] A spherical calibration object is provided, which can be detected simultaneously by a thermopile array and a depth camera, with the center of the spherical calibration object as a feature point;

[0010] The spherical calibration object is positioned such that the temperature field observed by the thermopile array has central symmetry;

[0011] Change the spatial pose of the spherical calibration object and simultaneously acquire the first coordinate set of the sphere's center in the thermopile array coordinate system and the second coordinate set in the depth camera coordinate system;

[0012] The spatial transformation relationship between the thermopile array and the depth camera is solved based on the first coordinate set and the second coordinate set.

[0013] Optionally, the spherical calibrator is a solid sphere of aluminum alloy or copper alloy, with a surface coated with high emissivity blackbody paint and an emissivity ≥0.94.

[0014] Optionally, the spherical calibrator is embedded with a heating wire. A constant current is continuously passed through the sphere to raise its temperature and stabilize it within 5 minutes. The surface temperature of the sphere is uniform, and the relative deviation between the highest and lowest temperatures does not exceed 2%.

[0015] Optionally, the central symmetry location includes: the center of a single pixel in the thermopile image, the center of an adjacent 2×2 pixel, and the midpoint of the boundary between two adjacent pixels.

[0016] Optionally, the step of determining that the temperature field has central symmetry includes: extracting a local temperature matrix with the candidate pixel position as the center, calculating the centroid coordinates by treating the values ​​of the matrix elements as the weights of the corresponding mass points, and determining that the central symmetry is established when the deviation between the centroid coordinates and the coordinates of the central pixel position is less than a preset threshold.

[0017] Optionally, after placing the spherical calibration object in the candidate position, the process further includes a fine-tuning step: iteratively moving the spherical calibration object along the row and column directions of the thermopile image with a step size of less than 0.05 pixels until the symmetry error reaches its minimum and no longer decreases after three consecutive iterations.

[0018] Optionally, the size of the local temperature matrix is ​​adaptively selected according to the candidate location type: a 3×3 neighborhood is taken for the center of a single pixel, a 2×2 neighborhood is taken for the center of a 2×2 pixel, and a 3×2 or 2×3 neighborhood is taken for the midpoint of the boundary.

[0019] Optionally, the step of changing the spatial pose of the spherical calibration object includes: generating no less than 64 spatial points in a uniform distribution within a set travel range, and changing the distance between the thermopile array and the sphere every 16 points to improve the stability of the depth direction solution of the projection matrix.

[0020] Optionally, the spatial transformation relationship is obtained by solving the 3×4 projection matrix. When solving, the coordinates of the central symmetry position corresponding to the center of the sphere are used as the observation values ​​of the thermopile image, and the three-dimensional center coordinates output by the depth camera are used as the world coordinates.

[0021] Secondly, the present invention provides a joint calibration system for a thermopile array and a depth camera, used to implement the above method, comprising:

[0022] The spherical calibration object can be detected simultaneously by a thermopile array and a depth camera, with the center of the sphere as the feature point.

[0023] The pose adjustment device is used to position the spherical calibration object at a location that makes the temperature field observed by the thermopile array centrally symmetric, and to change the spatial pose of the spherical calibration object during the calibration process;

[0024] The synchronous acquisition module, connected to the thermopile array and the depth camera, is used to synchronously acquire the first coordinate set of the sphere center in the thermopile array coordinate system and the second coordinate set in the depth camera coordinate system at each pose.

[0025] The calculation unit is used to solve the spatial transformation relationship between the thermopile array and the depth camera based on the first coordinate set and the second coordinate set, and output the calibration results.

[0026] The beneficial effects of this invention are:

[0027] (1) The resolution of thermopile arrays is very low, such as 8×8, and the positioning error is large, which cannot meet the calibration accuracy requirements. Through this invention, the image coordinates of thermopile arrays do not depend on detection, but on aiming at a specific position. By utilizing the temperature measurement symmetry, the aiming accuracy is much higher than that of detection, thus obtaining high-precision thermopile array image coordinates.

[0028] (2) The spherical calibrator in this invention can support different sensors with large differences in detection angles, and detection from opposite directions is within the scope of this invention.

[0029] (3) The calibrator has built-in heating, which makes the calibrator detectable in both visible and infrared wavelength ranges. Attached Figure Description

[0030] Figure 1 Schematic diagram of the thermopile array and depth camera to be calibrated;

[0031] Figure 2 Schematic diagram of a joint calibration system for thermopile array and depth camera;

[0032] Figure 3 Schematic diagram of the calibration ball structure;

[0033] Figure 4 A schematic diagram showing the center of the sphere located at the pixel center in a thermopile image;

[0034] Figure 5 A schematic diagram showing the location of the center of the sphere at a pixel corner in a thermopile image;

[0035] Figure 6 A schematic diagram showing the center of a sphere located at the center of a pixel's edge in a thermopile image;

[0036] Figure 7 Another schematic diagram showing the center of a sphere located at the center of a pixel's edge in a thermopile image. Detailed Implementation

[0037] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0038] In this embodiment, the combined thermopile array and depth camera system to be calibrated is as follows: Figure 1 As shown, the depth camera is realized by binocular stereo vision composed of a first camera 1 and a second camera 2. The first thermopile array sensor 3, the second thermopile array sensor 4, and the third thermopile array sensor 5 measure temperature from different angles. All sensors have an overlapping field of view 6, which can be observed simultaneously through two modes: visible light image and infrared temperature measurement. The system supports system calibration with thermopile array sensors and cameras arranged at arbitrary angles, and supports high-precision geometric calibration based on low-resolution images of the thermopile array.

[0039] Furthermore, the depth camera in this embodiment can be any principle scheme, including binocular stereo vision, structured light, etc. It is assumed that the depth camera itself has been calibrated and can image the target and acquire distance information.

[0040] Furthermore, in this embodiment, the field of view of the first thermopile array sensor 3, the second thermopile array sensor 4, and the third thermopile array sensor 5 is approximately 19°×19°, and the output image resolution is 8×8; the output image resolution of the first camera 1 and the second camera 2 is 1280×800.

[0041] like Figure 2 As shown, during calibration, this embodiment introduces a spherical calibration object, specifically a small metal sphere 8. This small metal sphere 8 has a built-in heating unit, and its temperature differs from the ambient temperature after heating, thus allowing it to be detected simultaneously by the depth camera and the thermopile array sensor. The small metal sphere 8 appears as a circle in the thermopile or visible light camera at any angle, and its center is used as a calibration feature point, corresponding to the center of the small metal sphere 8, thereby achieving feature point matching for observations at any angle.

[0042] Furthermore, in a preferred embodiment, the metal ball 8 is fixed to the displacement stage 7 (i.e., the posture adjustment device) by a rigid support rod 9. The displacement stage 7 can move in three mutually perpendicular directions, thereby realizing arbitrary adjustment and change of the three-dimensional position of the metal ball 8.

[0043] Furthermore, the minimum step of the displacement stage 7 is less than 0.1 mm, and the stroke is 100 mm × 100 mm × 300 mm.

[0044] Furthermore, in a preferred example, such as Figure 3 As shown, the metal ball 8 is made of aluminum alloy or copper alloy, and its excellent thermal conductivity ensures the uniformity of the ball temperature. The diameter of the metal ball 8 is about 12mm, and the distance between it and the thermopile array sensor during calibration is about 100~300mm. Therefore, its size in the thermopile array image is about 1~3 pixels, and it can measure temperatures higher than the ambient temperature within a 2×2 or 3×3 pixel range.

[0045] A blind hole with a diameter of 5mm and a depth of 8mm is machined on one side of the metal ball 8; the support rod 9 is a stainless steel rod with a diameter of 2mm, with the heating wire 10 wrapped around the front end, and then inserted into the blind hole and filled with insulating glue 11 for fixation; the insulating glue layer fills the gap to ensure that the heat of the heating wire 10 is conducted to the metal ball 8, but its thermal resistance is much greater than that of the aluminum alloy of the metal ball 8, thereby avoiding uneven temperature of the metal ball 8 caused by uneven heating.

[0046] The heating wire 10 heats up after being connected to the DC power supply 12, causing the metal ball 8 to heat up. Once the temperature stabilizes, the metal ball 8 is hotter than the ambient temperature, making it observable by the thermopile array sensor. Adjusting the voltage of the DC power supply 12 regulates the power, thereby adjusting the stable temperature of the metal ball 8. By continuously passing a constant current, the ball can be heated and stabilized within 5 minutes, with uniform surface temperature and a relative deviation between the highest and lowest temperatures not exceeding 2%.

[0047] Furthermore, the surface of the metal sphere 8 is coated with black body paint 13 (emissivity ≥ 0.94) to increase the emissivity and ensure its uniformity, so that it appears as a uniform radiation source in the field of view of the thermopile array.

[0048] Each thermopile array sensor was calibrated individually. Before calibration, the support rod was adjusted so that it faced away from the thermopile array sensor to be calibrated, in order to avoid the radiation from the support rod affecting the observation effect of the thermopile array.

[0049] The calibration system of this application also includes:

[0050] The synchronous acquisition module, connected to the thermopile array and the depth camera, is used to synchronously acquire the first coordinate set of the sphere center in the thermopile array coordinate system and the second coordinate set in the depth camera coordinate system at each pose.

[0051] The calculation unit is used to solve the spatial transformation relationship between the thermopile array and the depth camera based on the first coordinate set and the second coordinate set, and output the calibration results.

[0052] Based on the above system structure, this application embodiment also provides a joint calibration method for a thermopile array and a depth camera, the method comprising:

[0053] Step 1. Provide a spherical calibration object, which can be detected simultaneously by a thermopile array and a depth camera, and use the center of the spherical calibration object as a feature point.

[0054] Step 2. Place the spherical calibration object at a position that makes the temperature field observed by the thermopile array have central symmetry.

[0055] In a preferred example, during calibration, the spherical calibration object is always placed at a position that satisfies the following requirement: when the center of the calibration object is located at that point, the local temperature field data observed by the thermopile has central symmetry, that is, any two symmetrical pixels centered on the center image point have the same temperature measurement value.

[0056] Furthermore, the pixel coordinates of the sphere center projected onto the thermopile array image can be one of the following three positions:

[0057] 1) The center of a certain pixel, such as Figure 4 As shown, during calibration, the center of the sphere is located at the pixel center in the thermopile image, and its coordinates can be represented as (i,j), where i and j are both integers;

[0058] 2) The center of a certain adjacent 2×2 pixel, such as Figure 5 As shown, during calibration, the center of the sphere is located at a pixel corner in the thermopile image, and its coordinates can be represented as (i+0.5,j+0.5), where i and j are both integers;

[0059] 3) The midpoint of the boundary between two adjacent pixels, such as Figure 6 or Figure 7 As shown, during calibration, the center of the sphere is located at the center of a certain edge of a pixel in the thermopile image, and its coordinates can be represented as (i+0.5,j) or (i,j+0.5), where i and j are both integers.

[0060] Furthermore, in this embodiment, the thermopile array image is 8×8 pixels, which can be optionally... Figure 4 There are a total of 6×6=36 locations shown, which can be selected. Figure 5 There are a total of 7×7=49 locations shown, which can be selected. Figure 6 The locations shown total 6×7×2=84; a total of 169 thermopile image coordinates.

[0061] In the calibration process of this application, the center of the sphere is required to be located at these positions in the thermopile image because the pixel resolution of the thermopile image is very low. If the image coordinates of the center of the sphere are directly detected based on conventional methods, the error will be very large, generally exceeding 0.2 pixels. However, if the center of the sphere is located at the aforementioned positions, it can be verified by temperature measurement symmetry; generally, only a small deviation from the center of symmetry is needed to significantly disrupt the temperature measurement symmetry, which can then be detected. Therefore, this technique transforms the relatively difficult problem of "detecting the absolute coordinates of the center of the sphere at any position" into the relatively simple problem of "determining whether the center of the sphere is located at the center of symmetry," thereby significantly improving the positioning accuracy and solving the problem of excessive calibration errors caused by the low resolution of thermopile images.

[0062] Step 3. Change the spatial pose of the spherical calibration object and simultaneously acquire the first coordinate set of the sphere's center in the thermopile array coordinate system and the second coordinate set in the depth camera coordinate system.

[0063] In a preferred example, the position of the ball is adjusted multiple times during the calibration process so that the ball is at different positions and distances in the field of view of the thermopile array. Multiple sets of coordinate pairs of the ball center in the thermopile array and the depth camera are collected and recorded to form the first coordinate set and the second coordinate set.

[0064] Furthermore, in this embodiment, a total of 64 sets of calibration ball positions are recorded. During calibration, the distance between the ball and the thermopile array is changed every 16 sets of positions, which can improve the solution quality of the calibration results.

[0065] Furthermore, the method to position the center of the calibration sphere at a specific symmetrical point is as follows: place the sphere on an automatic or manual displacement stage, then continuously fine-tune the displacement stage, acquire thermopile images after each fine-tuning, extract local temperature measurement data centered on the symmetrical point (candidate pixel position), treat the temperature measurement value of each pixel as the mass weight of the corresponding position, calculate the centroid coordinates of the local temperature measurement data, and then calculate the distance between the centroid and the symmetrical point position. Adjust the displacement stage to reduce the distance until the distance is less than the set upper limit, or until the distance no longer decreases after multiple consecutive adjustments.

[0066] Furthermore, the statement that the distance no longer decreases after multiple consecutive adjustments specifically means: after placing the spherical calibration object in the candidate position, iteratively move the spherical calibration object along the row and column directions of the thermopile image with a step size of less than 0.05 pixels until the symmetry error reaches its minimum and no longer decreases after three consecutive iterations.

[0067] Furthermore, the local temperature measurement data used to calculate the center of gravity in this embodiment is as follows:

[0068] 1) If the center of the sphere is located in the thermopile image as follows Figure 4 As shown, its coordinates can be represented as (i,j), where i and j are both integers; then, the centroid is calculated by taking the data of a 3×3 matrix from column i-1 to column i+1 and row j-1 to row j+1.

[0069] 2) If the center of the sphere is located in the thermopile image as follows: Figure 5 As shown, its coordinates can be represented as (i+0.5, j+0.5), where i and j are both integers; then, the centroid is calculated by taking the data of a 2×2 matrix from column i to column i+1 and row j to row j+1.

[0070] 3) If the center of the sphere is located in the thermopile image as follows: Figure 6 As shown, its coordinates can be represented as (i+0.5,j), where i and j are both integers; then, the centroid is calculated by taking the data of the 3×2 matrix from column i to i+1 and row j-1 to j+1.

[0071] 4) If the center of the sphere is located in the thermopile image as follows: Figure 7 As shown, its coordinates can be represented as (i,j+0.5), where i and j are both integers; then, the centroid is calculated by taking the data of the 2×3 matrix from column (i-1) to column (i+1) and row (j) to column (j+1).

[0072] Furthermore, the method for calculating the centroid in this embodiment is as follows: assuming the pixel set is {(u k ,v k If k=1,2,…}, then the centroid (u cent ,v cent )for:

[0073]

[0074]

[0075] Step 4. Solve the spatial transformation relationship between the thermopile array and the depth camera based on the first coordinate set and the second coordinate set.

[0076] In this embodiment, the transformation relationship between the thermopile array pixel coordinates and the camera coordinate system is calculated and represented by the projection matrix M.

[0077] Assume that the N sets of depth camera 3D homogeneous coordinates and thermopile image 2D homogeneous coordinates are respectively... and If k=1,2,……N, then the two sets of coordinate mapping relationships described by the 3×4 projection matrix M are as follows:

[0078]

[0079] in It is only used to normalize the third element of the coordinates of a homogeneous image.

[0080] Let the 12 elements of matrix M be labeled row by row as m1, m2...m 12 Then we have:

[0081]

[0082]

[0083] The calibration record and Substituting the coordinates into the above equation, each set of coordinates can establish two linear equations about the elements of matrix M; therefore, at least 6 sets of coordinate pairs are needed to solve for matrix M. In this embodiment, the projection matrix M is obtained by solving an overdetermined system of linear equations based on 64 sets of coordinate pairs.

[0084] The various embodiments described in this specification are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0085] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A joint calibration method for a thermopile array and a depth camera, characterized in that: A spherical calibration object is provided, which can be detected simultaneously by a thermopile array and a depth camera, with the center of the spherical calibration object as a feature point; The spherical calibration object is positioned such that the temperature field observed by the thermopile array has central symmetry; Change the spatial pose of the spherical calibration object and simultaneously acquire the first coordinate set of the sphere's center in the thermopile array coordinate system and the second coordinate set in the depth camera coordinate system; The spatial transformation relationship between the thermopile array and the depth camera is solved based on the first coordinate set and the second coordinate set.

2. The joint calibration method for a thermopile array and a depth camera according to claim 1, characterized in that: The spherical calibrator is a solid sphere of aluminum alloy or copper alloy, with a surface coated with high emissivity blackbody paint, and an emissivity ≥0.

94.

3. A joint calibration method for a thermopile array and a depth camera according to claim 1 or 2, characterized in that: The spherical calibration object has an embedded heating wire. A constant current can be continuously passed through the sphere to raise its temperature and stabilize it within 5 minutes. The surface temperature of the sphere is uniform, and the relative deviation between the highest and lowest temperatures does not exceed 2%.

4. The joint calibration method for a thermopile array and a depth camera according to claim 1, characterized in that: The central symmetry locations include: the center of a single pixel in the thermopile image, the center of an adjacent 2×2 pixel, and the midpoint of the boundary between two adjacent pixels.

5. The joint calibration method for a thermopile array and a depth camera according to claim 4, characterized in that: The steps for determining that the temperature field has central symmetry include: extracting a local temperature matrix with the candidate pixel position as the center, calculating the centroid coordinates by treating the values ​​of the matrix elements as the weights of the corresponding mass points, and determining that the central symmetry is valid when the deviation between the centroid coordinates and the candidate pixel position coordinates is less than a preset threshold.

6. The joint calibration method for a thermopile array and a depth camera according to claim 5, characterized in that: After placing the spherical calibration object in the candidate position, the process further includes a fine-tuning step: iteratively moving the spherical calibration object along the row and column directions of the thermopile image with a step size of less than 0.05 pixels until the symmetry error reaches its minimum and no longer decreases after three consecutive iterations.

7. The joint calibration method for a thermopile array and a depth camera according to claim 5, characterized in that: The size of the local temperature matrix is ​​adaptively selected according to the candidate location type: the center of a single pixel takes a 3×3 neighborhood, the center of a 2×2 pixel takes a 2×2 neighborhood, and the midpoint of the boundary takes a 3×2 or 2×3 neighborhood.

8. The joint calibration method for a thermopile array and a depth camera according to claim 1, characterized in that: The steps of changing the spatial pose of the spherical calibration object include: generating no less than 64 spatial points evenly distributed within a set travel range, and changing the distance between the thermopile array and the sphere every 16 points to improve the stability of the depth direction solution of the projection matrix.

9. The joint calibration method for a thermopile array and a depth camera according to claim 1, characterized in that: The spatial transformation relationship is obtained by solving the 3×4 projection matrix. When solving, the coordinates of the centrally symmetric position corresponding to the center of the sphere are used as the observation values ​​of the thermopile image, and the three-dimensional center coordinates output by the depth camera are used as the world coordinates.

10. A joint calibration system for a thermopile array and a depth camera, used to implement the method according to any one of claims 1-9, characterized in that, include: The spherical calibration object can be detected simultaneously by a thermopile array and a depth camera, with the center of the sphere as the feature point. The pose adjustment device is used to position the spherical calibration object at a location that makes the temperature field observed by the thermopile array centrally symmetric, and to change the spatial pose of the spherical calibration object during the calibration process; The synchronous acquisition module, connected to the thermopile array and the depth camera, is used to synchronously acquire the first coordinate set of the sphere center in the thermopile array coordinate system and the second coordinate set in the depth camera coordinate system at each pose. The calculation unit is used to solve the spatial transformation relationship between the thermopile array and the depth camera based on the first coordinate set and the second coordinate set, and output the calibration results.

Images