Time-of-flight camera calibration method with probability and reflection analysis

Abstract

The invention relates to a method for calibrating time-of-flight cameras using basic probability and analyzing the light reflectance properties of objects.

Description

[0001] TIME-OF-FLIGHT CAMERA CALIBRATION METHOD WITH PROBABILITY AND REFLECTION ANALYSIS

[0002] Technical Field

[0003] The invention relates to a method of calibrating time-of-flight cameras by probability and reflectance analysis.

[0004] More particularly, the invention relates to a method for calibrating time-of-flight cameras using elementary probability and analyzing the light reflection properties of objects.

[0005] Prior Art

[0006] Cameras emerged as analog devices, and the process of image formation was expressed through geometric formulas. The parameters included in these formulas, which are still mostly valid for today's digital cameras, are called intrinsic and extrinsic camera parameters, and the process of obtaining these parameters is called camera calibration. Among the many different methods of camera calibration, the most widely used method today is the Zhang method. (https: / / www.youtube.com / watch7v-9He7Nu3u8s).

[0007] Time-of-flight cameras (ToF) stand out as a widely used type of camera in range imaging. Today, these devices are mostly found as sensor systems that share the same base with an RGB camera.

[0008] Camera calibration has been studied in many areas such as data fusion, optimization and image smoothing, and the capabilities of time-of-flight cameras have been examined. Calibration of such devices is the most important step before image acquisition to ensure accurate measurements. It is usually not possible to find a general method for such devices as they contain different types of noise and often involve complex procedures. The distance errors of such devices are generally defined as systematic errors and linear, polynomial, B-Splines, constant value table and stereo pair vision methods are used to model these errors. For measurement verification, stereo vision systems, laser measurements and measurements at specified distances were used.

[0009] Calibration of time-of-flight cameras usually consists of two stages. Since time-of- flight cameras can be modelled as pinhole cameras, the first stage involves the classical methods used for RGB cameras. In this way, the distance images are undistorted with the calibration parameters of the ToF camera. In the second stage, errors are modelled to correct the distance errors.

[0010] CN111047650A discloses a calibration method using a checkered pattern plane for the calibration of time-of-flight cameras.

[0011] WO2017196690A1 discloses a calibration method using a checkered pattern plane and different patterns for the calibration of time-of-flight cameras.

[0012] When the known applications in the art are examined, it is necessary to develop a new method for the calibration of time-of-flight cameras with higher calibration performance.

[0013] Objectives of the Invention

[0014] The object of the present invention is to develop a method for calibration of time- of-flight cameras using basic probability and analyzing the light reflectance properties of objects with high calibration performance.

[0015] The inventive calibration method consists of two stages. In the first stage, vertexbased photogrammetric calibration is performed using Zhang's method and camera parameters for RGB and time-of-flight cameras are found and the relative positions of the cameras are determined. By correcting the distortions in the images, distance measurements are also indirectly corrected.

[0016] The second stage is mostly based on finding a function that represents the error as a function of distance. Based on this, an error model is proposed, taking into account environmental conditions and time-dependent measurement variations. In this model, measurement errors are defined for each pixel with conditional probabilities that vary with distance. The errors found during measurement can take different values for different measurements. Since these errors can be expressed by a normal distribution, measurements were made at different distances and the normal distribution parameter values of the errors at these distances were found. By fitting a function that passes through these parameter values, functions of normal distribution parameters that vary with distance were found. For a better calibration, errors due to lighting and other environmental conditions were also investigated and reflectance analysis was performed. Measurement planes were used in the calculation phase and ground truth data were obtained from these planes. Separate values were found for each pixel and a pixel-based table of constant values was created.

[0017] Detailed Description of the Invention

[0018] The figures related to the calibration method performed to achieve the object of this invention are explained below.

[0019] Figure 1: Schematic view of the checkerboard patterned plane used in the inventive method.

[0020] Figure 2: Schematic view of the plane used for reflectance analysis in the inventive method, where the first rectangle is black, the last rectangle is white, and each column contains a total of six rectangular areas in different shades of gray in 20% decreasing amounts. The parts in the figures are numbered and the equivalents of these numbers are given below.

[0021] 1. Checkerboard-patterned plane

[0022] 2. Plane used for reflection analysis

[0023] 3. Black region

[0024] 4. White region

[0025] 5. First gray region

[0026] 6. Second gray region

[0027] 7. Third gray region

[0028] 8. F ourth gr ay r egi on

[0029] M- Multiplier term representing reflection-induced errors

[0030] The inventive method; comprises,

[0031] Obtaining distance images using the checkerboard-patterned plane (1) and the plane used for reflection analysis (2), which includes 6 different shades ranging from black to white, for every 25 centimeters of distance between 0.5 meters and 3.5 meters,

[0032] - Determining the intrinsic parameters of the RGB and IR cameras, Correcting each of the distance images using the IR camera parameters,

[0033] - Defining a rectangular working area that also includes the center of the distance image and fitting a plane to the relevant 3D point cloud for each distance,

[0034] Assuming this fitted plane as the precise reference data for the relevant distance and determining the errors by comparing the measurements at this distance with the plane,

[0035] Assuming that the errors obtained from repeated measurements at the same distance in the previous step follow a normal distribution, and determining the values of the normal distribution parameters for each pixel at each measured distance,

[0036] - Determining second-degree regression models that define the values of the normal distribution parameters for each pixel depending on the distance,

[0037] - Using images of the plane used for reflection analysis (2), which includes 6 different shades ranging from black to white, taken from different distances to find the multiplier term representing reflection-induced errors (M), and determining the average value at each measured distance for each gray area,

[0038] - Dividing the average values found in the previous step by the exact reference values at the corresponding distances and determining the values at the measured distances for each region,

[0039] - Representing the division results obtained in the previous step according to distance and gray level, fitting a surface model that optimally expresses these values, and finding a function that defines the M term expressed by this model.

[0040] The inventive calibration method also consists of two steps. In the first step, vertexbased photogrammetric calibration is performed using Zhang's method and camera parameters for RGB and time-of-flight cameras are found and the relative positions of the cameras are determined. In the second step, the plane used for reflectance analysis (2), which includes 6 different color tones going from black to white, is used to find a function that shows the errors depending on the distance.

[0041] In photogrammetric calibration, using projective principles, the lens properties of cameras can be described by the pinhole camera model. In order to obtain accurate photogrammetric measurements, it is necessary to find the variables (calibration parameters of the cameras) of the formulas based on this model. The most common method used for camera calibration is to establish relationships between two- dimensional image coordinates and three-dimensional space coordinates using a well-defined planar pattern of known dimensions. After the distance image has been corrected after photogrammetric calibration, the distance camera must be calibrated to correct the distance measurements. Errors in distance measurements are not only due to manufacturing errors but also to environmental conditions such as light, heat and reflectivity of objects. These types of errors are classified as systematic and non-systematic.

[0042] The most common systematic error in such cameras is the wiggling error, which is caused by the phase difference measurement and is observed as a fluctuation of the values. These errors are usually observed as a sinusoidal curve depending on the distance and a polynomial function is sought to correct them. Although errors related to the integration time can be considered as systematic errors, these effects are usually ignored by choosing a fixed value.

[0043] Different types of noise such as thermal noise, quantization error, photon shot noise and scattering generated noise introduce random errors in distance measurements. Since the errors due to quantization noise and photon shot noise are usually very small, these errors are neglected. When the camera is switched on, internal heat builds up, which can affect distance measurements. Due to the CMOS architecture, pixels may be subject to energy spillover from neighboring pixels, called internal scattering. Depending on the manufacturing characteristics of the camera, this can be observed as discontinuities at the edges of the object. Thermal noise and noise due to internal energy scattering can be defined together as a general type of noise and can be modeled with a normal distribution.

[0044] In addition to these error sources, there are also problems called "multipath interference". Such errors are usually observed at the edges of the image and are caused by the interference of the IR reflections of surrounding objects. A similar effect can be seen when pixels corresponding to object edges receive light from both the front and back sides. In addition, a pixel can also be affected by possible manufacturing defects and physical failures over time. For this reason, repeated measurements of an object may vary. To minimize such effects, a pixel-based calibration method would be appropriate.

[0045] The reflective properties of the imaged objects can also affect the measurements. In studies, it has been observed that white paper reflects all of the 900 nm wavelength light, while a black car tire reflects two percent of the available light. The light energy received by the sensors generally varies depending on the reflective properties of the object and its position relative to the camera. Low- intensity measurements may contain random errors due to the low-power signal, and therefore less reflected light can lead to closer than normal measurements. Researchers have also found that the reflective properties of other objects in the environment also affect the measurements.

[0046] Distance measurements contain various errors due to the above-mentioned types of errors, and systematic bias and random errors are often used as the two main sources of error in modeling these errors. Considering that systematic errors also contain random features, the measurements can be modeled simply as follows. dr= dm+ e (1)

[0047] Where dris the actual distance, dmis the measured distance and e is the measurement error.

[0048] Depending on various internal and external factors, the errors in the distance measurement of a given point can vary continuously over a period of operation. Therefore, a random variable can be defined that maps measurements to error values. While it is possible to define a stochastic process using the full set of random variables defined for all pixels in a single image and perform further analysis on this basis, a simpler alternative to determine error values is to use a probability distribution to model them. The measurements and corresponding errors at each pixel of a distance image can be described as a distribution of random variables D and E, respectively. The error value of a single pixel can usually be described as a normal probability distribution. The error distribution can be defined as the posterior probability of distance measurements and can be expressed as

[0049] P [ E (p) | D ] ~ / V ( e, u2) (2)

[0050] Where E(p), p is the measurement pixel error, D is the measured distance, N ( e, u2) is the normal distribution of the error values in the measurements at pixel p. When different objects at a fixed distance are measured under constant conditions, different results can be obtained depending on the reflectivity of the objects. Similarly, the amount of ambient light and the light intensity on the object can also affect distance measurements. To express these effects in the above formula, a multiplier term, M, can be added to the formula and as a result the error distribution can be expressed as follows.

[0051] P [ E (p) | D ] ~ M * W ( e, ff2) (3)

[0052] To estimate the values of the M and normal distribution parameters in this formula, measurements of a target plane positioned perpendicular to the camera are obtained for different distances. After an interpolation using the values found for each distance, a distance-dependent function is determined and the values of these parameters are found. While an ordinary checkerboard-patterned plane (1) can be used to determine both the IR camera parameters and the normal distribution of error values, a modified planar pattern shown in Figure 2 is needed to find the values of the M term. In this model, there are six rectangular areas of different shades of gray, starting from 100% black and decreasing by 20% in each column. The plane used for the reflectance analysis (2) contains a black region (3) on the far left and a white region (4) on the far right. The first gray region (5), the second gray region (6), the third gray region (7) and the fourth gray region (8) form a plane with different shades of gray in each column, decreasing by 20% from black to white. Using measurements of the RGB and IR intensity values of these areas for different distances, an interpolation will be performed to estimate the value of this term based on distance.

[0053] To estimate the values of both the normal distribution and the normal distribution term parameters, measurements of a target plane positioned perpendicular to the camera are obtained for different distances and an interpolation is performed to estimate these values. While an ordinary checkerboard pattern can be used to determine both the IR camera parameters and the normal distribution of error values, a specially designed planar pattern, shown in Figure 3, is needed to find the values of the M term. In this pattern there are six rectangular areas of different dark colors, starting from 100% black and decreasing by 20% in each column. Using the measurements of the RGB and IR intensity values of these areas for different distances, an interpolation is performed to find the values of the M term depending on the distance.

[0054] In order to find errors in measurement values, ground truth values are needed for comparison. External systems such as 3D laser scanners or stereo cameras are often used to obtain accurate reference data. In this study, a variant of the RANSAC method is applied using known positions in the planar pattern used in photogrammetric calibration, and the position of this plane relative to the camera is determined and these values are considered as accurate reference data.

[0055] The distance camera calibration method proposed in this paper uses distance camera parameters, where the probability distribution and reflectance analysis terms are defined as a function of distance. These parameters are determined for each pixel and in this study, they are calculated for a region of interest located in the center of the sample data.

[0056] In order to find the normal distribution parameters of a pixel, errors in a possibly large number of measurement data for a given distance are detected. These errors are almost always normally distributed. This process is repeated for different distances and a regression model is fitted to express the normal distribution parameter values found. Using this model, the distance-dependent normal distribution parameter values for each pixel in the study region are found. In order to correct the errors due to reflectance, the reflectance values of the dark-colored areas in the planar pattern in Figure 2 are examined and a function expressing the values of the relevant term depending on the distance is found.

[0057] After these calculations, two functions were identified that express the normal distribution parameters (mean value e and variance u2) for each pixel in the study area, depending on the distance. In addition, a function expressing the M term as a function of distance and light intensity (darkness, grayness) was found. Using these functions, distance errors can be corrected for pixels in the defined study region and more accurate measurements can be made. By defining larger working regions, a table of constant values can be found that includes the values of all pixels. The method defined here corrects common errors in distance measurements.

Claims

CLAIMS1. A method for the calibration of time-of-flight cameras, characterized in that it comprises;Obtaining distance images using the checkerboard-patterned plane (1) and the plane used for reflection analysis (2), which includes 6 different shades ranging from black to white, for every 25 centimeters of distance between 0.5 meters and 3.5 meters,- Determining the intrinsic parameters of the RGB and IR cameras, Correcting each of the distance images using the IR camera parameters,- Defining a rectangular working area that also includes the center of the distance image and fitting a plane to the relevant 3D point cloud for each distance,Assuming this fitted plane as the precise reference data for the relevant distance and determining the errors by comparing the measurements at this distance with the plane,Assuming that the errors obtained from repeated measurements at the same distance in the previous step follow a normal distribution, and determining the values of the normal distribution parameters for each pixel at each measured distance,- Determining second-degree regression models that define the values of the normal distribution parameters for each pixel depending on the distance,- Using images of the plane used for reflection analysis (2), which includes 6 different shades ranging from black to white, taken from different distances to find the multiplier term representing reflection-induced errors (M), and determining the average value at each measured distance for each gray area,- Dividing the average values found in the previous step by the exact reference values at the corresponding distances and determining the values at the measured distances for each region,- Representing the division results obtained in the previous step according to distance and gray level, fitting a surface model that optimally expresses these values, and finding a function that defines the M term expressed by this model.

Images