Calibration method and device of iToF camera, and electronic equipment

Abstract

The application relates to a kind of iToF camera calibration method and device, electronic equipment.The method comprises: obtaining the real phase of each pixel point in different position image to the image sensor of iToF camera;The center wiggling value of the center point is obtained according to the measured phase and real phase of the center point, preset;In the first image, find the first pixel point whose measured phase is equal to the measured phase of the center point by iteration, and obtain its fppn value;In the second image, the second pixel point whose pixel coordinates are same with the pixel coordinates of the first pixel point is found by iteration, and its second wiggling value is obtained;When it is judged that the iteration meets the preset iteration convergence condition, the iteration operation is ended, and the calibration of iToF camera is completed.The application uses the checkerboard based on lens calibration as calibration board, and finds the pixel point by iteration, so that only two calibration board images in different positions are needed, the time required for calibration is reduced, and the calibration cost is saved.

Description

Technical Field

[0001] This invention relates to the field of ToF ranging technology, and in particular to a calibration method and apparatus for an iToF camera, as well as an electronic device. Background Technology

[0002] Indirect Time-of-Flight (iToF) refers to the indirect measurement of the flight time of light by measuring phase shift. For example... Figure 1 As shown, the iToF imaging principle is as follows: The iToF camera controls the light-emitting module 12 to actively emit modulated light signals through the modulation module 11; the emitted light is emitted onto the surface of the target object 19, and the reflected light signal formed after reflection by the target object 19 is sampled by the photosensitive pixel array unit 13 of the image sensor; then, the distance to the target object is calculated based on the phase shift of the emitted and reflected light. The light-emitting module 12, such as a VCSEL, an infrared emitter, or an LED, is usually driven by a modulated square wave generated by the image sensor. However, as the modulation frequency increases, the light waveform gradually approaches a sine wave, and the higher harmonics in the square wave will introduce periodic errors into the measurement, such as... Figure 2 As shown.

[0003] Because there is a certain deviation between the actual optical waveform and the ideal optical waveform, such as the presence of aliasing harmonics in the relevant waveforms leading to wiggling errors during the measurement process, for example... Figure 3As shown; and due to differences between pixels, there are certain deviations in the measurement of image sensors. For example, due to circuit delays, the photosensitive pixel array units in the image sensor will have a certain phase delay, resulting in Fixed Phase Pattern Noise (FPPN) errors. These deviations can usually be corrected on the production line through calibration, but wiggling calibration and FPPN calibration require special calibration boards and relatively complex calibration steps, which increases the calibration cost of iToF cameras. For example, by moving the calibration board on the guide rail and establishing an error lookup table based on the measured phase and the true phase, the wiggle error can be corrected well in principle; however, due to the influence of noise, multiple distance measurements are required, and each measurement needs to be averaged multiple times to effectively remove random noise, which increases the time cost of moving the calibration board and the time cost of obtaining multiple averages, thus increasing the calibration cost. While using an internal delay circuit to generate a virtual real distance saves the time cost of moving the calibration board, it still requires obtaining multiple average values ​​for each virtual real distance to reduce the impact of noise. Furthermore, the above method still cannot solve the problem of increased calibration costs caused by switching calibration boards on the production line.

[0004] Chinese patent publication CN113281726A discloses an error calibration method that uses an internal delay circuit to simulate and replace the traditional moving calibration board to generate a series of virtual real distances. Error calibration is then performed based on the measured distance and the real distance, thus completing the entire cycle of wiggling calibration. However, this disclosure does not consider the increased calibration cost caused by acquiring multiple average values ​​for each real distance to reduce the impact of noise. Furthermore, the deviation introduced by the internal delay circuit at high modulation frequencies can also affect the relevant accuracy requirements.

[0005] Therefore, how to reduce the time required for iToF camera calibration and save calibration costs is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] The purpose of this invention is to provide a calibration method, apparatus, and electronic device for iToF cameras, which solves the technical problems of long calibration time and high calibration cost of existing iToF cameras. By using the checkerboard pattern used for lens calibration as the calibration board, and simultaneously calibrating wiggling and FPPN, the calibration time required for iToF cameras is reduced and the calibration cost is saved.

[0007] To achieve the above objectives, the present invention provides a calibration method for an iToF camera, comprising the following steps: based on a first image of a calibration board at a first position in the field of view of the iToF camera, acquiring the true phase of each pixel in the first image to the image sensor of the iToF camera; and based on a second image of the calibration board at a second position in the field of view of the iToF camera, acquiring the true phase of each pixel in the second image to the image sensor of the iToF camera, wherein the calibration board is a checkerboard used for lens calibration, and the second position is parallel to the first position; the center fppn value of the center point of the first image is preset to 0, and the true phase of each pixel in the second image is acquired based on the measured phase and the true phase of the calibration board at a second position in the field of view of the iToF camera. The iToF camera calibration is performed by: 1) determining the center wiggling value of the center point; 2) iterating through the first image to find a first pixel whose measured phase is equal to the measured phase of the center point, and obtaining the first fppn value of the first pixel based on the measured phase, the true phase, and the center wiggling value; 3) iterating through the second image to find a second pixel whose pixel coordinates are the same as the first pixel, and obtaining the second wiggling value of the second pixel based on the measured phase, the true phase, and the first fppn value; and 4) ending the iterative traversal operation when a preset iterative convergence condition is met, thus completing the iToF camera calibration.

[0008] Optionally, the method further includes: when it is determined that the iterative traversal does not meet the preset iterative convergence condition, continuing to iteratively traverse the second image to find new pixels and obtain the corresponding fppn value based on the measured phase of the second pixel and the second wiggling value; and further ending the iterative traversal operation and completing the iToF camera calibration when it is determined that the iterative traversal meets the preset iterative convergence condition, and continuing to iteratively traverse the first image to find new pixels and obtain the corresponding wiggling value and the corresponding fppn value based on the pixel coordinates of the new pixels found in the second image and the obtained corresponding fppn value when it is determined that the iterative traversal does not meet the preset iterative convergence condition.

[0009] To achieve the above objectives, the present invention also provides a calibration device for an iToF camera, comprising: a first acquisition module, configured to acquire the true phase of each pixel in a first image of a calibration board at a first position in the field of view of the iToF camera, based on a first image captured by the camera, and to acquire the true phase of each pixel in a second image of the calibration board at a second position in the field of view of the iToF camera, based on a second image captured by the camera, at a second position, based on a second image of the calibration board, at a second position, based on a second image captured by the camera, at a second position, at a second position, at a second position, at a second position, at a second position, at a second position, at a second position, at a second position, at a second position, at a second position, at a second position, at a second position, at a second position, at a second position, at a second position, at a second position, at a second position, at a second position, at a second position, at a second position, at a third ... The system comprises: a center wiggling value; a first traversal module, used to traverse and find a first pixel in the first image whose measured phase is equal to the measured phase of the center point, and obtain a first fppn value of the first pixel based on the measured phase and the true phase corresponding to the first pixel and the center wiggling value; a second traversal module, used to iteratively traverse and find a second pixel in the second image whose pixel coordinates are the same as the pixel coordinates of the first pixel, and obtain a second wiggling value corresponding to the second pixel based on the measured phase and the true phase corresponding to the second pixel and the first fppn value; and a processing module, used to end the iterative traversal operation and complete the iToF camera calibration when it is determined that the iterative traversal meets a preset iterative convergence condition.

[0010] Optionally, the processing module is further configured to, when it is determined that the iterative traversal does not meet the preset iterative convergence condition, call the second traversal module to continue iteratively traversing the second image to find new pixels and obtain the corresponding fppn value based on the measured phase of the second pixel and the second wiggling value; the processing module is further configured to, when it is determined that the iterative traversal meets the preset iterative convergence condition, end the iterative traversal operation and complete the iToF camera calibration, and when it is determined that the iterative traversal does not meet the preset iterative convergence condition, call the first traversal module to, based on the pixel coordinates of the new pixels found in the second image and the obtained corresponding fppn value, continue iteratively traversing the first image to find new pixels and obtain the corresponding wiggling value and the corresponding fppn value.

[0011] To achieve the above objectives, the present invention also provides an electronic device, including a memory, a processor, and a computer-executable program stored in the memory and executable on the processor, wherein the processor executes the computer-executable program to implement the steps of the iToF camera calibration method as described in the present invention.

[0012] This invention uses a black and white checkerboard pattern, similar to that used in lens calibration, as a calibration board for wiggling and FPPN calibration. This eliminates the need for a separate white board covering the entire field of view for FPPN calibration, allowing for calibration on the production line without switching between different calibration board patterns, thus saving calibration costs while completing the calibration process. By using a checkerboard pattern based on lens calibration combined with an iterative pixel-finding method, this invention only requires acquiring two calibration board images at different locations, significantly reducing the number of calibration board movements and the time required for averaging multiple images. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of the iToF imaging principle;

[0015] Figure 2 The periodic error is caused by higher harmonics;

[0016] Figure 3 This refers to the oscillation error present during the measurement process;

[0017] Figure 4 This is a flowchart of an iToF camera calibration method according to an embodiment of the present invention;

[0018] Figure 5 This is a schematic diagram illustrating the acquisition of a chessboard pattern image according to an embodiment of the present invention;

[0019] Figure 6 This is a schematic diagram of an iterative traversal operation provided in an embodiment of the present invention;

[0020] Figure 7 This is a structural block diagram of an iToF camera calibration device provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] One embodiment of the present invention provides a calibration method for a ToF camera.

[0023] Please refer to the following: Figures 4-6 ,in, Figure 4 This is a flowchart of an iToF camera calibration method according to an embodiment of the present invention. Figure 5 This is a schematic diagram illustrating the acquisition of a chessboard pattern image according to an embodiment of the present invention. Figure 6 This is a schematic diagram of an iterative traversal operation provided in an embodiment of the present invention.

[0024] like Figure 4 As shown, the method described in this embodiment includes the following steps: S1, based on a first image of the calibration board at a first position in the field of view of the iToF camera, obtain the true phase of each pixel in the first image to the image sensor of the iToF camera, and based on a second image of the calibration board at a second position in the field of view of the iToF camera, obtain the true phase of each pixel in the second image to the image sensor of the iToF camera; S2, preset the center fppn value of the center point of the first image to 0, and obtain the center wiggling value of the center point according to the measured phase and the true phase of the center point; S3, traverse the first image... S4. Find a first pixel whose measured phase is equal to the measured phase of the center point, and obtain the first fppn value of the first pixel based on the measured phase and the true phase corresponding to the first pixel and the center wiggling value; S5. Iteratively traverse the second image to find a second pixel whose pixel coordinates are the same as the pixel coordinates of the first pixel, and obtain the second wiggling value corresponding to the second pixel based on the measured phase and the true phase corresponding to the second pixel and the first fppn value; and S6. When it is determined that the iterative traversal meets the preset iterative convergence condition, end the iterative traversal operation and complete the calibration of the iToF camera.

[0025] Regarding step S1, based on the first image of the calibration board at the first position in the field of view of the iToF camera, the true phase of each pixel in the first image to the image sensor of the iToF camera is obtained; and based on the second image of the calibration board at the second position in the field of view of the iToF camera, the true phase of each pixel in the second image to the image sensor of the iToF camera is obtained. The checkerboard pattern used for lens calibration is used as the calibration board, and the second position is parallel to the first position. Specifically, the iToF camera 51 captures an image of the checkerboard pattern used for lens calibration, which serves as the calibration board 52, to obtain an image of the checkerboard pattern in the field of view (Fov1) of the iToF camera, as shown below. Figure 5 As shown. Translating the chessboard grid allows you to obtain images from two different positions; for example, obtaining image A at the first position and image B at the second position, as shown. Figure 6 As shown.

[0026] In image measurement and machine vision applications, to determine the relationship between points on the surface of a spatial object and image coordinates, a geometric model of camera imaging needs to be established. These geometric model parameters are the camera parameters. Camera parameters mainly include intrinsic and extrinsic parameter matrices, distortion coefficients, and focal length. The intrinsic parameter matrix is ​​determined by the camera's internal parameters, while the extrinsic parameter matrix is ​​determined by the relative position of the camera and the target object (pixel coordinates are rotated and translated to coincide with world coordinates; this rotation and translation matrix is ​​the extrinsic parameter matrix). The extrinsic parameter matrix is ​​a parameter related to the true position. Typically, the intrinsic parameter matrix is ​​calibrated first, and then the extrinsic parameter matrix is ​​calculated based on the intrinsic parameter matrix to calculate the true distance.

[0027] The calibration plane of the calibration board is the surface of the checkerboard facing the lens of the iToF camera, that is, the plane in the checkerboard used to reflect the light emitted by the iToF camera. In this embodiment, by using the black and white checkerboard used in lens calibration as the calibration board for wiggling and fppn calibration, there is no need to provide an additional white board covering the entire field of view for fppn calibration. This eliminates the need to switch between different calibration board patterns during production line calibration, saving calibration costs while completing the calibration. By using the checkerboard based on lens calibration as the calibration board, combined with the subsequent iterative traversal to find pixels, only two calibration board images at different positions need to be obtained, greatly reducing the number of times the calibration board needs to be moved and reducing the time for averaging multiple images.

[0028] In some embodiments, step S1 further includes: (11) performing lens calibration using the checkerboard as a calibration board to determine the intrinsic parameter matrix of the iToF camera; (12) capturing a first image of the calibration board at a first position in the field of view of the iToF camera based on the intrinsic parameter matrix to determine the extrinsic parameter matrix of the iToF camera; (13) obtaining the true distance from each pixel in the first image to the image sensor based on the intrinsic parameter matrix and the extrinsic parameter matrix; and (14) obtaining the true phase from each pixel in the first image to the image sensor based on the true distance. Wherein, the checkerboard has a first mounting tilt angle θ at the first position, such as... Figure 5 As shown. In other embodiments, the intrinsic and extrinsic parameter matrices of the iToF camera can also be obtained using the Zhang Zhengyou calibration method. Obtaining the true phase from each pixel in the second image to the iToF camera's image sensor can be done in the same way as obtaining the true phase from each pixel in the first image to the iToF camera's image sensor, and will not be elaborated here. The obtained true phases of each pixel can also be stored in a table for easy lookup of the corresponding true phase based on the pixel.

[0029] Following the above embodiment, the step of obtaining the true distance from each pixel in the first image to the image sensor based on the intrinsic parameter matrix and the extrinsic parameter matrix further includes: converting the pixel coordinates to world coordinates using a coordinate transformation formula based on the intrinsic parameter matrix and the extrinsic parameter matrix; and calculating the true distance using a distance calculation formula.

[0030] Specifically, the following coordinate transformation formula is used to convert pixel coordinates to world coordinates:

[0031]

[0032] Among them, z c M1 represents the scaling factor, u and v are the coordinates of a pixel in the pixel coordinate system, the matrix corresponding to M1 is the intrinsic parameter matrix, u0 and v0 are the center points of the intrinsic parameter matrix, and f x and f y Let M1 be the focal length of the intrinsic parameter matrix, M2 be the matrix corresponding to the extrinsic parameter matrix, r be the direction vector of the pixel coordinate system's coordinate axes in the world coordinate system's coordinate axes, t be the translation vector from the origin of the world coordinate system to the origin of the pixel coordinate system, and x be the direction vector of the pixel coordinate system's coordinate axes in the world coordinate system's coordinate axes. w y w z w These are the coordinates of the corresponding point in the world coordinate system.

[0033] Specifically, the true distance d is calculated using the following distance calculation formula:

[0034] d = sqrt((x) w *x w )+(y w *x w )+(z w *z w )).

[0035] Following the above embodiment, the step of obtaining the true phase from each pixel in the first image to the image sensor based on the true distance further includes: calculating the true phase phase_real using the following formula:

[0036]

[0037] Where phase_real is the real phase, d is the real distance, f is the modulation frequency of the emitted light, and phase_max is one phase period (i.e., 2π).

[0038] Regarding step S2, the center fppn value of the center point of the first image is preset to 0. The center wiggling value of the center point is obtained based on the measured phase and the true phase of the center point. Specifically, when the preset fppn value of the center point (for ease of distinction, it is denoted as the center fppn value in this embodiment) is 0, the wiggling value of the center point (for ease of distinction, it is denoted as the center wiggling value in this embodiment) can be obtained based on the known measured phase and true phase of the center point.

[0039] In some embodiments, the center wiggling value of the center point is calculated using the following formula:

[0040] phase_real=wiggling_lut[phase_measure]-fppn i，j (Formula 1)

[0041] Among them, fppn i，j Here, fppn is the fppn value of the pixel, phase_real is the real phase of the pixel, wiggling_lut is the wiggling value of the pixel, and phase_measure is the measured phase of the pixel output by the iToF camera. At this point, the pixel is the center point of the first image, hence fppn... i，j =0.

[0042] Regarding step S3, the process involves traversing the first image to find a first pixel whose measured phase is equal to the measured phase of the center point, and obtaining the first fppn value of the first pixel based on the measured phase, the true phase, and the center wiggling value. Specifically, there is at least one pixel in the first image whose measured phase is equal to the measured phase of the center point. Therefore, by traversing the first image based on the measured phase of the center point, all pixels whose measured phase is equal to the measured phase of the center point can be found (for ease of distinction, in this embodiment, they are referred to as the first pixel).

[0043] In some embodiments, the fppn value of the corresponding first pixel is calculated using Formula 1 above. Since the measured phase of each found first pixel is equal to the measured phase of the center point, its wiggling value is the same as the center wiggling value; therefore, based on the known measured phase, wiggling value and true phase of each first pixel, the fppn value of the corresponding pixel can be calculated using Formula 1 above.

[0044] Regarding step S4, iteratively traverse the second image to find a second pixel point having the same pixel coordinate as the first pixel point, and obtain a second wiggling value corresponding to the second pixel point according to the measured phase and real phase corresponding to the second pixel point and the first fppn value. Specifically, for pixel points with the same pixel coordinates in two images (the first image and the second image), their fppn values are also the same; therefore, based on the known measured phase, real phase and fppn value of each second pixel point, the wiggling value of the corresponding pixel point can be calculated by the foregoing Formula 1.

[0045] Regarding step S5, when it is determined that the iterative traversal satisfies a preset iterative convergence condition, the iterative traversal operation is ended, and the calibration of the iToF camera is completed. Specifically, when the pixel points found in the first image and the obtained wiggling values and fppn values, in combination with the pixel points found in the second image and the obtained wiggling values and fppn values, satisfy the calibration accuracy of the iToF camera, the iterative traversal operation can be ended, and the calibration of the iToF camera is completed.

[0046] In some embodiments, the iterative convergence condition is:

[0047] abs(find_pixel_cnt-width*height)<thr (Formula 2)

[0048] wherein find_pixel_cnt is the count value of the found pixel points, width and height are respectively the width and height of the image resolution of the iToF camera, and thr is a preset threshold. That is, when the absolute value of the difference between the total number of pixel points found in the first image and the second image and the image resolution is less than the preset threshold, the iterative traversal operation is ended, and the calibration of the iToF camera is completed.

[0049] In some embodiments, the method further includes: S6, when it is determined that the iterative traversal does not meet a preset iterative convergence condition, continuing to iteratively traverse the second image to find new pixels and obtain corresponding fppn values ​​based on the measured phase of the second pixel and the second wiggling value; and S7, further ending the iterative traversal operation and completing the iToF camera calibration when it is determined that the iterative traversal meets a preset iterative convergence condition, and continuing to iteratively traverse the first image to find new pixels and obtain corresponding wiggling values ​​and corresponding fppn values ​​based on the pixel coordinates of the new pixels found in the second image and the obtained corresponding fppn values ​​when it is determined that the iterative traversal does not meet the preset iterative convergence condition. Specifically, if the pixel values ​​and their corresponding wiggling and fppn values ​​found in the first image, combined with those found in the second image, do not meet the calibration accuracy requirements of the iToF camera, the iterative traversal operation can continue to be performed alternately between the second and first images until the preset iterative convergence condition is met. It should be noted that after each iterative traversal operation, a check is performed to determine whether the iterative traversal meets the preset iterative convergence condition. If it does, the iterative traversal operation ends; otherwise, iterative traversal continues.

[0050] Following the above embodiment, the step S6, which involves iteratively traversing the second image to find new pixels and obtain the corresponding fppn value, further includes: iteratively traversing the second image to find third pixels whose measured phase is equal to the measured phase of the second pixel, and obtaining the third fppn value of the third pixel based on the measured phase, the true phase, and the second wiggling value. Specifically, since the measured phase of each found third pixel is equal to the measured phase of the second pixel, its wiggling value is the same as the wiggling value of the second pixel; therefore, based on the known measured phase, wiggling value, and true phase of each third pixel, the fppn value of the corresponding pixel can be calculated using the above formula 1. After the iterative traversal operation described in step S6 is completed, the iterative traversal operation ends when it is determined that the iterative traversal meets the preset iterative convergence condition, completing the iToF camera calibration, and step S7 is no longer executed; if it is determined that the iterative traversal does not meet the preset iterative convergence condition, step S7 continues to be executed.

[0051] Following the above embodiment, the step S7 of iteratively searching for new pixels in the first image and obtaining the corresponding wiggling value and fppn value further includes: (71) iteratively searching for a fourth pixel in the first image whose pixel coordinates are the same as those of the third pixel, and obtaining the fourth wiggling value corresponding to the fourth pixel based on the measured phase and true phase of the fourth pixel and the third fppn value, and ending the iterative traversal operation when the preset iterative convergence condition is met, thus completing the calibration of the iToF camera. Specifically, since pixels with the same pixel coordinates in the two images (the first image and the second image) also have the same fppn value; therefore, based on the known measured phase, true phase and fppn value of each fourth pixel, the wiggling value of the corresponding pixel can be calculated using the above formula 1. After the iterative traversal operation described in step (71) is completed, the iterative traversal operation ends when it is determined that the iterative traversal meets the preset iterative convergence condition, the iToF camera calibration is completed, and the subsequent step (72) is no longer executed; if it is determined that the iterative traversal does not meet the preset iterative convergence condition, the subsequent step (72) continues to be executed.

[0052] Following the above embodiment, the step S7 of continuing to iteratively traverse the first image to find new pixels and then obtain the corresponding wiggling value and the corresponding fppn value further includes: (72) when it is determined that the iterative traversal does not meet the preset iterative convergence condition, iteratively traversing the first image to find a fifth pixel whose measured phase is equal to the measured phase of the fourth pixel, and obtaining the fifth fppn value of the fifth pixel based on the measured phase and the true phase corresponding to the fifth pixel and the fourth wiggling value, and ending the iterative traversal operation when the preset iterative convergence condition is met, thus completing the calibration of the iToF camera. Specifically, since the measured phase of each found fifth pixel is equal to the measured phase of the fourth pixel, its wiggling value is the same as the wiggling value of the fourth pixel; therefore, based on the known measured phase, wiggling value and true phase of each fifth pixel, the fppn value of the corresponding pixel can be calculated using the above formula 1. After the iterative traversal operation described in step (72) is completed, the iterative traversal operation ends when it is determined that the iterative traversal meets the preset iterative convergence condition, thus completing the iToF camera calibration. If it is determined that the iterative traversal does not meet the preset iterative convergence condition, the iterative traversal continues in the second image to find new pixels and obtain the corresponding wiggling value and the corresponding fppn value, based on the pixel coordinates of the new pixel point (fifth pixel point) found in the first image and the corresponding fppn value obtained. For specific operation methods, please refer to the aforementioned steps S4 to S6, which will not be repeated here.

[0053] The following combination Figure 6 The iterative traversal operation of this invention will be further explained. Assume that in the illustration, A is the first image at a first position, and B is the second image at a second position; A0 is the center point of the first image, with its center fppn value preset to 0. The center wiggling value of the center point is calculated using the formula 1 above. Only a portion of the found pixels are shown in the figure for illustrative purposes.

[0054] The specific iterative traversal process is as follows:

[0055] 1) Traverse the first image A to find the pixel points whose measured phase is equal to the measured phase of the center point A0. For example, find n pixels from A11 to A1n. These n pixels are all recorded as the first pixel point.

[0056] 2) Based on the measured phase and true phase corresponding to A11 to A1n and the center wiggling value, the first fppn value corresponding to A11, the first fppn value corresponding to A12, ..., the first fppn value corresponding to A1n can be calculated respectively using the above formula 1; at the same time, each of these n first pixel points A11 to A1n corresponds to a pixel coordinate.

[0057] 3) Iteratively traverse the second image to find pixels whose pixel coordinates are the same as those of A11 to A1n. For example, find k1 pixels (B111 to B11k1) whose pixel coordinates are the same as those of A11, find k2 pixels (B121 to B12k2) whose pixel coordinates are the same as those of A12, ..., find pixels (B1n1 to B1nk) whose pixel coordinates are the same as those of A12. n Total k n The pixels whose pixel coordinates are the same as those of A1n, k1, k2, ..., k n Each pixel is recorded as the second pixel.

[0058] 4) Based on the measured phase and true phase corresponding to B111~B11k1 and the first fppn value corresponding to A11, obtain the second wiggling values ​​corresponding to B111~B11k1 respectively; based on the measured phase and true phase corresponding to B121~B12k2 and the first fppn value corresponding to A12, obtain the second wiggling values ​​corresponding to B121~B12k2 respectively; ..., based on B1n1~B1nk n The corresponding measured phase and true phase, as well as the first fppn value corresponding to A1n, are used to obtain B1n1 to B1nk respectively. n The corresponding second wiggling value;

[0059] 5) Determine whether the iterative convergence condition shown in Formula 2 is met. If it is met, end the iterative traversal operation and complete the iToF camera calibration. Otherwise, continue to execute the subsequent steps.

[0060] 6) Iteratively traverse the second image to find the measured phases that are equal to B111~B11k1, B121~B12k2, ..., B1n1~B1nk respectively. n The pixels whose measurement phase is the same as B1111 to B111p1 are identified. For example, find p1 pixels whose measurement phase is the same as B111, p2 pixels whose measurement phase is the same as B112, p2 pixels whose measurement phase is the same as B112, ..., find pixels whose measurement phase is the same as B11k11 to B11k1p1. k Total p kFind q1 pixels whose measurement phase is the same as the measurement phase of B11k1, from B1211 to B111q1, whose measurement phase is the same as the measurement phase of B121; find q2 pixels whose measurement phase is the same as the measurement phase of B122, from B1212 to B112q2; ..., find pixels from B12k21 to B12k2q... k Total q k Find r1 pixels whose measurement phase is the same as the measurement phase of B12k2, ..., find r1 pixels whose measurement phase is the same as the measurement phase of B1n1 (B1n11 to B1n1r1), find r2 pixels whose measurement phase is the same as the measurement phase of B1n2 (B1n21 to B1n2r2), ..., find r1 pixels whose measurement phase is the same as the measurement phase of B1n2, ..., find r1 pixels whose measurement phase is the same as the measurement phase of B1nk. n 1~B1nk n r k Total r k Each measurement phase and B1nk n The pixels with the same measurement phase, p1 to p2 k ,q1~q k ..., r1~r k Each pixel is recorded as the third pixel.

[0061] 7) Based on the measured phase and true phase corresponding to B1111~B111p1 and the wiggling value corresponding to B111, the third fppn value corresponding to B1111~B111p1 can be calculated using Formula 1 above. Similarly, based on the measured phase and true phase corresponding to B1121~B112p2 and the wiggling value corresponding to B112, the third fppn value corresponding to B1121~B112p2 can be calculated using Formula 1 above, and so on. Based on B1nk... n 1~B1nk n r k The corresponding measured phase and true phase, and B1nk n The corresponding Wiggling value can be calculated using Formula 1 above. n 1~B1nk n r k The corresponding third fppn value; at the same time, these p1~p k , q1~q k ..., r1~r k Each third pixel point corresponds to a pixel coordinate;

[0062] 8) Again check whether the iterative convergence condition shown in Formula 2 is met. If it is met, end the iterative traversal operation and complete the iToF camera calibration; otherwise, according to p1~p k , q1~q k ..., r1~r kThe pixel coordinates of the third pixel and the corresponding fppn value are obtained. Then, the image is iteratively traversed in the first image to find new pixels and obtain the corresponding wiggling value and the corresponding fppn value.

[0063] It should be noted that pixels with the same pixel coordinates found in the same image can be merged to improve traversal efficiency. For example, if B1111 and B1n1r1 have the same pixel coordinates, they can be merged, and the corresponding third fppn value can be obtained in only one calculation.

[0064] As can be seen from the above, this invention uses the black and white checkerboard pattern used in lens calibration as a calibration board for wiggling and FPPN calibration. This eliminates the need for an additional white board covering the entire field of view for FPPN calibration, thus avoiding the need to switch between different calibration board patterns during production line calibration, saving calibration costs while completing the calibration. By using a checkerboard pattern based on lens calibration as the calibration board, combined with subsequent iterative traversal to find pixels, this invention only requires acquiring calibration board images from two different positions, significantly reducing the number of calibration board movement steps and the time required for averaging multiple images.

[0065] Based on the same inventive concept, the present invention also provides a calibration device for an iToF camera. The provided iToF camera calibration device can employ, for example... Figure 4 The calibration method shown is for calibrating iToF cameras.

[0066] Please see Figure 7 This is a structural block diagram of an iToF camera calibration device provided in an embodiment of the present invention. Figure 7 As shown, the calibration device for the iToF camera includes: a first acquisition module 71, a second acquisition module 72, a first traversal module 73, a second traversal module 74, and a processing module 75.

[0067] Specifically, the first acquisition module 71 is used to acquire the true phase of each pixel in the first image (captured at a first position in the iToF camera's field of view) to the image sensor of the iToF camera, and to acquire the true phase of each pixel in the second image (captured at a second position in the iToF camera's field of view) to the image sensor of the iToF camera, based on the first image of the calibration board at a first position in the iToF camera's field of view. The calibration board is a checkerboard pattern used for lens calibration, and the second position is parallel to the first position. The second acquisition module 72 is used to acquire the center wiggling value of the center point when the center fppn value of the center point of the preset first image is 0, based on the measured phase and the true phase of the center point. The first traversal module 73 is used to traverse the first image to find a first pixel whose measured phase is equal to the measured phase of the center point, and to acquire the first fppn value of the first pixel based on the measured phase and the true phase corresponding to the first pixel, as well as the center wiggling value. The second traversal module 74 is used to iteratively traverse the second image to find a second pixel whose pixel coordinates are the same as those of the first pixel, and obtain the second wiggling value corresponding to the second pixel based on the measured phase and the true phase corresponding to the second pixel and the first fppn value. The processing module 75 is used to end the iterative traversal operation and complete the iToF camera calibration when it is determined that the iterative traversal meets the preset iterative convergence condition. The specific working methods of each module can be found in [reference needed]. Figure 2 The descriptions of the corresponding steps in the FPPN calibration method for the ToF camera shown are not repeated here.

[0068] In some embodiments, the processing module 75 is further configured to, when determining that the iterative traversal does not meet a preset iterative convergence condition, call the second traversal module 74 to continue iteratively traversing the second image to find new pixels and obtain corresponding fppn values ​​based on the measured phase of the second pixel and the second wiggling value. The processing module 75 is further configured to, when determining again that the iterative traversal meets the preset iterative convergence condition, end the iterative traversal operation and complete the iToF camera calibration; and when determining that the iterative traversal does not meet the preset iterative convergence condition, call the first traversal module 73 to, based on the pixel coordinates of the new pixels found in the second image and the obtained corresponding fppn values, continue iteratively traversing the first image to find new pixels and obtain corresponding wiggling and fppn values.

[0069] Based on the same inventive concept, the present invention also provides an electronic device, including a memory, a processor, and a computer-executable program stored in the memory and executable on the processor; when the processor executes the computer-executable program, it implements as follows: Figure 4 The steps of the iToF camera calibration method are shown.

[0070] Within the scope of this inventive concept, embodiments can be described and illustrated based on modules that perform one or more of the described functions. These modules (also referred to herein as units, etc.) can be physically implemented by analog and / or digital circuitry, such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, etc., and can optionally be driven by firmware and / or software. The circuitry can, for example, be implemented in one or more semiconductor chips. The circuitry constituting a module can be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware performing some functions of the module and a processor performing other functions of the module. Without departing from the scope of this inventive concept, each module of an embodiment can be physically divided into two or more interactive and discrete modules. Similarly, without departing from the scope of this inventive concept, the modules of an embodiment can be physically combined into more complex modules.

[0071] Generally, terms can be understood at least partially from their usage in context. For example, the term "one or more" as used herein depends at least in part on the context and can be used to describe a feature, structure, or characteristic in a singular sense, or in a plural sense to describe a combination of features, structures, or characteristics. Additionally, the term "based on" can be understood not necessarily to express an exclusive set of factors, but rather, alternatively, also depends at least in part on the context, allowing for the presence of other factors that are not necessarily explicitly described.

[0072] It should be noted that the terms "comprising" and "having," and their variations, used in this invention document are intended to cover non-exclusive inclusion. The terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence, unless explicitly indicated by the context. It should be understood that such data used interchangeably where appropriate. Furthermore, embodiments and features within embodiments of this invention can be combined with each other unless otherwise specified. In addition, descriptions of well-known components and technologies have been omitted in the above description to avoid unnecessarily obscuring the concepts of this invention. In the various embodiments described above, each embodiment focuses on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0073] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A calibration method for an iToF camera, characterized in that, Includes the following steps: Based on the first image of the calibration board at the first position in the field of view of the iToF camera, the true phase of each pixel in the first image to the image sensor of the iToF camera is obtained; and based on the second image of the calibration board at the second position in the field of view of the iToF camera, the true phase of each pixel in the second image to the image sensor of the iToF camera is obtained, wherein the checkerboard used for lens calibration is used as the calibration board, and the second position is parallel to the first position; The center fppn value of the center point of the first image is preset to 0. Based on the measured phase and the true phase of the center point, the center wiggling value of the center point is obtained, and the following formula is used to calculate the center wiggling value of the center point: phase_real = wiggling_lut[phase_measure] fppn i，j (Formula 1), where fppn i，j Here, fppn is the fpn value of the pixel, phase_real is the real phase of the pixel, wiggling_lut is the wiggling value of the pixel, phase_measure is the measured phase of the pixel output by the iToF camera, and the pixel is the center point of the first image. i，j =0; In the first image, the first pixel point whose measured phase is equal to the measured phase of the center point is searched through the image. The wiggling value of each first pixel point is the same as the wiggling value of the center point. Based on the known measured phase and true phase of the first pixel point and the wiggling value of the center point, the first fppn value of the first pixel point can be calculated by formula 1. In the second image, the second pixel point with the same pixel coordinates as the first pixel point is iteratively searched. The fppn values ​​of the pixels with the same pixel coordinates in the first image and the second image are the same. Based on the known measured phase and true phase corresponding to the second pixel point and the first fppn value, the second wiggling value corresponding to the second pixel point can be calculated by the formula 1. The iterative traversal operation ends when the preset iterative convergence condition is met, thus completing the calibration of the iToF camera.

2. The method according to claim 1, characterized in that, The step of obtaining the true phase of each pixel in the first image from the calibration board at the first position in the field of view of the iToF camera to the image sensor of the iToF camera further includes: Lens calibration is performed using the chessboard as a calibration board to determine the intrinsic parameter matrix of the iToF camera; Based on the intrinsic parameter matrix, a first image of the calibration board at a first position in the field of view of the iToF camera is captured to determine the extrinsic parameter matrix of the iToF camera; wherein, the checkerboard grid has a first mounting tilt angle at the first position; Based on the intrinsic parameter matrix and the extrinsic parameter matrix, obtain the true distance from each pixel in the first image to the image sensor; and Based on the true distance, the true phase of each pixel in the first image to the image sensor is obtained.

3. The method according to claim 2, characterized in that, The step of obtaining the true distance from each pixel in the first image to the image sensor based on the intrinsic parameter matrix and the extrinsic parameter matrix further includes: Based on the intrinsic and extrinsic parameter matrices, the pixel coordinates are converted to world coordinates using the following coordinate transformation formula: ， ， ， Among them, z c M1 represents the scaling factor, u and v are the coordinates of a pixel in the pixel coordinate system, the matrix corresponding to M1 is the intrinsic parameter matrix, u0 and v0 are the center points of the intrinsic parameter matrix, and f x and f y Let M1 be the focal length of the intrinsic parameter matrix, M2 be the matrix corresponding to the extrinsic parameter matrix, r be the direction vector of the pixel coordinate system's coordinate axes in the world coordinate system's coordinate axes, t be the translation vector from the origin of the world coordinate system to the origin of the pixel coordinate system, and x be the direction vector of the pixel coordinate system's coordinate axes in the world coordinate system's coordinate axes. w y w z w The coordinates of the corresponding point in the world coordinate system; and The true distance d is calculated using the following distance calculation formula: 。 4. The method according to claim 2, characterized in that, The step of obtaining the true phase from each pixel in the first image to the image sensor based on the true distance further includes: The true phase is calculated using the following formula: ， Where phase_real is the real phase, d is the real distance, f is the modulation frequency of the emitted light, and phase_max is one phase period.

5. The method according to claim 1, characterized in that, The iterative convergence condition is: ， Where find_pixel_cnt is the count of the found pixels, width and height are the width and height of the iToF camera's image resolution, respectively, and thr is the preset threshold.

6. The method according to claim 1, characterized in that, The method further includes: When it is determined that the iterative traversal does not meet the preset iterative convergence condition, based on the measured phase of the second pixel and the second wiggling value, the iterative traversal in the second image continues to search for new pixels and obtain the corresponding fppn value; and Further, when it is determined that the iterative traversal meets the preset iterative convergence condition, the iterative traversal operation ends and the iToF camera calibration is completed. When it is determined that the iterative traversal does not meet the preset iterative convergence condition, based on the pixel coordinates of the new pixel found in the second image and the obtained corresponding fppn value, the iterative traversal in the first image continues to search for new pixels and then obtain the corresponding wiggling value and the corresponding fppn value.

7. The method according to claim 6, characterized in that, The step of iteratively searching for new pixels in the second image and obtaining the corresponding fppn value further includes: iteratively searching for a third pixel in the second image whose measured phase is equal to the measured phase of the second pixel, and obtaining the third fppn value of the third pixel based on the measured phase and the true phase corresponding to the third pixel and the second wiggling value; The step of iteratively searching for new pixels in the first image and obtaining the corresponding wiggling value and the corresponding fppn value further includes: iteratively searching for a fourth pixel in the first image whose pixel coordinates are the same as those of the third pixel; obtaining the fourth wiggling value corresponding to the fourth pixel based on the measured phase and the true phase corresponding to the fourth pixel and the third fppn value; and ending the iterative traversal operation when the preset iterative convergence condition is met, thus completing the calibration of the iToF camera. When it is determined that the iterative traversal does not meet the preset iterative convergence condition, the fifth pixel point with a measured phase equal to the measured phase of the fourth pixel point is iteratively traversed in the first image. The fifth fppn value of the fifth pixel point is obtained based on the measured phase and the true phase corresponding to the fifth pixel point, as well as the fourth wiggling value. The iterative traversal operation ends when the preset iterative convergence condition is met, and the calibration of the iToF camera is completed.

8. A calibration device for an iToF camera, characterized in that, include: The first acquisition module is used to acquire the true phase of each pixel in the first image of the calibration board at a first position in the field of view of the iToF camera, and to acquire the true phase of each pixel in the second image of the calibration board at a second position in the field of view of the iToF camera, based on the first image of the calibration board at a first position in the field of view of the iToF camera, based on the first image of the calibration board at a second position in the field of view of the iToF camera, based on the second image of the calibration board at a second position in the field of view of the iToF camera, wherein the checkerboard used for lens calibration is used as the calibration board, and the second position is parallel to the first position; The second acquisition module is used to, when the center fppn value of the center point of the preset first image is 0, acquire the center wiggling value of the center point based on the measured phase and the true phase of the center point, and calculate the center wiggling value of the center point using the following formula: phase_real = wiggling_lut[phase_measure] fppn i，j (Formula 1), where fppn i，j Here, fppn is the fpn value of the pixel, phase_real is the real phase of the pixel, wiggling_lut is the wiggling value of the pixel, phase_measure is the measured phase of the pixel output by the iToF camera, and the pixel is the center point of the first image. i，j =0; The first traversal module is used to traverse and find the first pixel in the first image whose measured phase is equal to the measured phase of the center point. The wiggling value of each found first pixel is the same as the center wiggling value. Based on the known measured phase and true phase of the first pixel and the center wiggling value, the first fppn value of the first pixel can be calculated using the formula 1. The second traversal module is used to iteratively traverse the second image to find a second pixel whose pixel coordinates are the same as those of the first pixel. Pixels with the same pixel coordinates in both the first and second images have the same fppn value. Based on the known measured phase and true phase of the second pixel, and the first fppn value, the second wiggling value corresponding to the second pixel can be calculated using Formula 1. The processing module is used to end the iterative traversal operation and complete the iToF camera calibration when the iterative traversal meets the preset iterative convergence condition.

9. The apparatus according to claim 8, characterized in that, The processing module is further configured to, when it is determined that the iterative traversal does not meet the preset iterative convergence condition, call the second traversal module to continue iteratively traversing in the second image to find new pixels and obtain the corresponding fppn value based on the measured phase of the second pixel and the second wiggling value. The processing module is further configured to end the iterative traversal operation and complete the iToF camera calibration when it is determined that the iterative traversal meets the preset iterative convergence condition; and to call the first traversal module to continue iteratively traversing the first image to find new pixels and obtain the corresponding wiggling value and the corresponding fppn value based on the pixel coordinates of the new pixels found in the second image and the corresponding fppn value obtained.

10. An electronic device comprising a memory, a processor, and a computer-executable program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-executable program, it implements the steps of the iToF camera calibration method as described in any one of claims 1 to 7.

Images