Calibration method, calibration system, depth camera and readable storage medium

Abstract

This application provides a calibration method, a calibration system, a depth camera, and a readable storage medium. The calibration method is applied to a calibration system including a depth camera, a calibration board, and a controller and processor, and includes: calculating a first distance between the first and second intersection points of the depth camera under ideal conditions based on the camera coordinates of a first intersection point and a second intersection point and preset intrinsic and extrinsic parameters of the depth camera; calculating a second distance between the first and second intersection points based on the calculation principle of the depth camera and the echo signal; determining whether the image plane of the depth camera is parallel to the calibration board by comparing the first and second distances, and selecting whether to perform calibration correction on the depth camera based on the determination result (for example, when the image plane of the depth camera is tilted relative to the calibration board, the depth camera is in a non-ideal state, and calibration correction is performed on the depth camera). This application eliminates the adverse effects of tilt error and improves the measurement accuracy and precision of the depth camera.

Description

[Technical Field]

[0001] This application relates to the field of depth camera technology, and in particular to a calibration method, calibration system, depth camera, and readable storage medium. [Background Technology]

[0002] In the existing technology, due to the existence of systematic and random errors, the measurement results and accuracy of depth cameras are affected by many factors such as the internal and external environment of the camera. Therefore, in order to obtain more accurate depth information, it is necessary to calibrate the depth values ​​of the depth camera.

[0003] Currently, the industry standard for depth camera calibration is as follows: obtain the actual distance between the depth camera and the calibration position; move the calibration plate to the calibration position; measure the distance between the depth camera and the calibration plate to obtain the corresponding measured distance; compare the measured distance with the actual distance to obtain the calibration parameters, thereby completing the depth camera calibration.

[0004] However, the components in a depth camera may have tilt errors due to improper assembly. The calibration method for the depth camera mentioned above does not take this tilt error into account, resulting in inaccurate measurement results and low measurement accuracy.

[0005] Therefore, it is necessary to improve the above-mentioned calibration method for depth cameras. [Summary of the Invention]

[0006] This application provides a calibration method, calibration system, depth camera, and readable storage medium, aiming to solve the problem in the prior art that the measurement results of the depth camera are inaccurate and the measurement accuracy is low because the tilt error of the depth camera is not taken into account.

[0007] To address the aforementioned technical problems, a first aspect of this application provides a calibration method applied to a calibration system, which includes a depth camera, calibration plates, and a controller and processor. The calibration method includes: controlling the depth camera's transmitting module to emit the same light beam to a first and second calibration plate, which are at different distances and parallel to each other; and controlling the depth camera's acquisition module to receive echo signals reflected back from each calibration plate; wherein the light beam has a first intersection point and a second intersection point with the first and second calibration plates, respectively; acquiring the camera coordinates of each intersection point and calculating a first distance between each intersection point using preset intrinsic and extrinsic parameters of the depth camera; wherein, ideally, the image plane of the depth camera is parallel to the calibration plates; calculating a second distance between the first and second intersection points based on the echo signals and the depth camera's calculation principle; determining whether the image plane of the depth camera is parallel to the calibration plates based on the first and second distances, and selecting whether calibration correction of the depth camera is needed based on the determination result.

[0008] The second aspect of this application provides a calibration system, including a guide rail, a calibration plate slidably connected to the guide rail, a base, a depth camera, and a control and processor, wherein: the depth camera is placed on the base, the base and the calibration plate are respectively set at both ends of the guide rail, the control and processor is used to control the calibration plate to slide on the guide rail, control the transmitting module of the depth camera to emit light signals to the calibration plate and control the acquisition module to receive echo signals reflected back from the calibration plate at different distances, and execute the calibration method described in the first aspect of this application according to the received echo signals to complete the calibration of the depth camera.

[0009] A third aspect of this application provides a depth camera, including a projection module, an acquisition module, a processing module, and a storage module. The storage module stores calibration parameters acquired during the calibration method described in the first aspect of this application. The projection module projects light signals onto a target area. The acquisition module receives echo signals reflected back from the target area. The processing module generates a depth image of the target area based on the reflected echo signals and corrects the depth image based on the calibration parameters in the storage module to obtain a corrected depth image.

[0010] A fourth aspect of this application provides a computer-readable storage medium storing executable instructions, which, when executed, perform the calibration method described in the first aspect of this application.

[0011] As can be seen from the above description, compared with the prior art, the beneficial effects of this application are as follows: the depth camera emits the same light beam to the first calibration plate and the second calibration plate which are at different distances and parallel to each other, and receives the echo signal reflected back by the calibration plate, wherein the same light beam emitted by the depth camera has a first intersection point with the first calibration plate and a second intersection point with the second calibration plate. When calibrating the depth camera, this application first calculates a first distance (for reference) between the first and second intersection points under ideal conditions (indicating that the image plane of the depth camera is parallel to the calibration plate) based on the camera coordinates of the first and second intersection points and the preset intrinsic and extrinsic parameters of the depth camera. Then, based on the calculation principle of the depth camera and the echo signal, a second distance (i.e., the actual distance) between the first and second intersection points is calculated. Finally, by comparing the first and second distances, it is determined whether the image plane of the depth camera is parallel to the calibration plate, and based on the determination result, it is selected whether the depth camera needs to be calibrated (for example, if the image plane of the depth camera is parallel to the calibration plate, it means that the depth camera is currently in an ideal state and calibration is not necessary; if the image plane of the depth camera is tilted relative to the calibration plate, it means that the depth camera is currently in a non-ideal state and calibration is necessary). In short, this application fully considers the tilt error (i.e., non-ideal state) of the depth camera when calibrating it. When the depth camera has a tilt error, this application will further calibrate and correct the depth camera, thereby eliminating the adverse effects of the tilt error to a large extent, and thus effectively improving the measurement accuracy of the depth camera and the accuracy of the measurement results. [Attached Image Description]

[0012] To more clearly illustrate the related technologies or the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the related technologies or the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application, and not all embodiments. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A schematic flowchart illustrating the calibration method provided in the embodiments of this application;

[0014] Figure 2 The optical path diagram of the calibration method provided in the embodiments of this application during the calibration process;

[0015] Figure 3 A calibration effect diagram of the calibration method provided in the embodiments of this application at 1000mm.

[0016] Figure 4A calibration effect diagram of the calibration method provided in the embodiments of this application at 2000mm.

[0017] Figure 5 A block diagram of a depth camera provided in an embodiment of this application;

[0018] Figure 6 A block diagram of a computer-readable storage medium provided in an embodiment of this application. [Specific implementation method]

[0019] To make the objectives, technical solutions, and advantages of this application more apparent and understandable, the application will be clearly and completely described below in conjunction with the embodiments and corresponding drawings. Throughout, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. It should be understood that the various embodiments of this application described below are merely illustrative and not intended to limit the application. That is, all other embodiments obtained by those skilled in the art based on the various embodiments of this application without creative effort are within the scope of protection of this application. Furthermore, the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0020] Figure 1 This is a flowchart illustrating the calibration method provided in an embodiment of this application. The calibration method is applied to a depth camera, and the application environment of this calibration method is as follows: Figure 2 As shown: a first calibration plate B1 and a second calibration plate B2 are mounted on a guide rail (not shown) and slide in conjunction with the guide rail; a base (not shown) is located at one end of the guide rail; a depth camera (not shown) is mounted on the base; the surfaces of the first calibration plate B1 and the second calibration plate B2 are perpendicular to the extension direction of the guide rail; the same light beam G emitted by the depth camera has a first intersection point A1 with the first calibration plate B1 and a second intersection point A2 with the second calibration plate B2; the first calibration plate B1 is parallel to the second calibration plate B2; the distances between the first calibration plate B1, the second calibration plate B2 and the depth camera are not the same; the number of calibration plates is not limited to two (i.e., the first calibration plate B1 and the second calibration plate B2); it can be one calibration plate (i.e., the first calibration plate B1 moves on the guide rail to obtain the second calibration plate B2), or it can include three or more; the description here is only an example.

[0021] In the above application environment, all points on the same light beam G have the same corresponding pixel on the image plane (i.e., M1 or M2) of the depth camera. That is, ideally, after reflection by the first calibration plate B1 and the second calibration plate B2, the same light beam G will illuminate pixel P on M1 (i.e., ideally, the corresponding pixels on M1 for the first intersection point A1 and the second intersection point A2 are both P). Non-ideally, after reflection by the first calibration plate B1 and the second calibration plate B2, the same light beam G will illuminate pixel P′ on M2 (i.e., non-ideally, the corresponding pixels on M2 for the first intersection point A1 and the second intersection point A2 are both P). The corresponding pixels are all P′). However, since the non-ideal state has a tilt error compared to the ideal state, the corresponding pixels P of the first intersection point A1 and the second intersection point A2 on M1 in the ideal state are different from the corresponding pixels P′ of the first intersection point A1 and the second intersection point A2 on M2 in the non-ideal state. However, the actual distance between the first intersection point A1 and the second intersection point A2 in the external space is independent of whether the image plane of the depth camera is tilted. That is, the actual distance between the first intersection point A1 and the second intersection point A2 in the external space is always the Euclidean distance between the first intersection point A1 and the second intersection point A2.

[0022] Specifically, the calibration method provided in this application includes the following steps 101 to 105.

[0023] Step 101: Control the depth camera's transmitting module to emit the same beam to the first and second calibration plates, which are at different distances and parallel to each other, and control the depth camera's acquisition module to receive the echo signal reflected back by the calibration plates.

[0024] In this embodiment, when calibrating the depth camera, it is necessary to control the depth camera to emit the same light beam G to a first calibration plate B1 and a second calibration plate B2 that are at different distances and parallel to each other, and to receive the echo signals reflected back by each calibration plate. The intersection point of the same light beam G with the first calibration plate B1 is the first intersection point A1, and the intersection point with the second calibration plate B2 is the second intersection point A2. It should be noted that the same light beam G indicates that the light beam emitted by the depth camera's emission module to the first calibration plate B1 and the second calibration plate B2 is a single beam of light in the same direction; it can also be multiple beams of light in different directions. When it is multiple beams of light in different directions, the intersection points are calculated sequentially according to the direction of each beam, and no limitation is imposed here.

[0025] Step 102: Obtain the camera coordinates of the intersection point, and calculate the first distance between the first and second intersection points of the depth camera under ideal conditions based on the camera coordinates of the first and second intersection points and the preset intrinsic and extrinsic parameters of the depth camera.

[0026] In this embodiment, the camera coordinates of the intersection points are obtained, and the camera coordinates of the two intersection points are obtained based on the world coordinates of the first intersection point A1 and the second intersection point A2 and the preset intrinsic and extrinsic parameters of the depth camera. Furthermore, a first distance between the first intersection point A1 and the second intersection point A2 is calculated. Since this distance is calculated through coordinate transformation, it can be considered as the first distance under ideal conditions, where ideal conditions indicate that the image plane of the depth camera is parallel to the first calibration plate B1 and the second calibration plate B2. Figure 2 In this context, M1 represents the image plane of the depth camera under ideal conditions, and the first distance serves as a reference.

[0027] Step 103: Calculate the second distance between the first intersection point and the second intersection point based on the calculation principle of the depth camera and the echo signal.

[0028] In this embodiment, the distance between the first intersection point A1 and the second intersection point A2, obtained based on the calculation principle of the depth camera, is the second distance. This second distance is calculated based on the echo signals received by the depth camera and reflected by each calibration plate. It should be noted that whether the depth camera is in an ideal state is unknown when calculating the second distance. Therefore, this distance obtained based on the depth camera's calculation principle is the actual distance, intended for comparison with the first distance used as a reference in step 102.

[0029] Step 104: Compare the first distance and the second distance, and determine whether the image plane of the depth camera is parallel to the calibration plate.

[0030] In this embodiment, after obtaining the first distance and the second distance, it is also necessary to determine whether the image plane of the depth camera is parallel to the first calibration plate B1 and the second calibration plate B2 based on the obtained first and second distances, which is equivalent to determining whether the depth camera is currently in an ideal state or a non-ideal state. As an example, if the first distance and the second distance are the same, then it is determined that the image plane of the depth camera is parallel to the first calibration plate B1 and the second calibration plate B2, i.e., the depth camera is currently in an ideal state; if the first distance and the second distance are different, then it is determined that the image plane of the depth camera is tilted relative to the first calibration plate B1 and the second calibration plate B2, i.e., the depth camera is currently in a non-ideal state, where a non-ideal state indicates that the image plane of the depth camera is tilted relative to the first calibration plate B1 and the second calibration plate B2. Figure 2 M2 in the text represents the image plane of the depth camera in a non-ideal state. In a non-ideal state, the depth camera has a tilt error because the image plane of the depth camera is tilted relative to the first calibration plate B1 and the second calibration plate B2.

[0031] Step 105: Based on the judgment result, select whether the depth camera needs to be calibrated.

[0032] In this embodiment, after determining whether the image plane of the depth camera is parallel to the first calibration plate B1 and the second calibration plate B2, it is necessary to select whether the depth camera needs to be calibrated and corrected based on the determination result. In this context, calibration and correction refers to correcting the tilt error of the depth camera to eliminate its adverse effects. For example, if the image plane of the depth camera is parallel to the first calibration plate B1 and the second calibration plate B2, meaning the depth camera is currently in an ideal state, then the output result indicates that no calibration and correction of the depth camera is required. If the image plane of the depth camera is tilted relative to the first calibration plate B1 and the second calibration plate B2, meaning the depth camera is currently in a non-ideal state, then calibration and correction of the depth camera is performed to eliminate the adverse effects of the tilt error.

[0033] In this embodiment, when calibrating a depth camera, the image plane of the depth camera is determined based on a first distance and a second distance to see if it is parallel to the first calibration plate B1 and the second calibration plate B2. Based on the determination result, it is selected whether calibration correction of the depth camera is necessary. For example, if the image plane of the depth camera is parallel to the first calibration plate B1 and the second calibration plate B2, it means that the depth camera is currently in an ideal state and calibration correction is not required. If the image plane of the depth camera is tilted relative to the first calibration plate B1 and the second calibration plate B2, it means that the depth camera is currently in a non-ideal state and calibration correction is required. In other words, this embodiment fully considers the tilt error (i.e., non-ideal state) of the depth camera during calibration. When the depth camera has a tilt error, this embodiment further calibrates and corrects the depth camera, thereby largely eliminating the adverse effects of the tilt error and effectively improving the measurement accuracy and the accuracy of the measurement results.

[0034] In one implementation, the "first distance" in step 102 is calculated based on the camera coordinate system, and may specifically include: calculating the first distance according to a first formula; wherein the first formula is expressed as:

[0035]

[0036] D(P) represents the first distance, u represents the x-coordinate of the pixel P on the image plane M1 of the depth camera corresponding to the first intersection point A1 and the second intersection point A2, v represents the y-coordinate of the pixel P on the image plane M1 of the depth camera corresponding to the first intersection point A1 and the second intersection point A2, (u,v) can be calculated based on the world coordinates of the intersection points and the preset extrinsic parameters of the depth camera, f represents the focal length of the depth camera, d1 represents the distance between the first calibration plate B1 and the depth camera, d2 represents the distance between the second calibration plate B2 and the depth camera, and c xc represents the x-coordinate of the center of the image plane M1 of the depth camera under ideal conditions. y This represents the center ordinate of the image plane M1 of the depth camera under ideal conditions.

[0037] This implementation corresponds to the ideal state, where the first distance is easy to obtain. It is only necessary to first slide the first calibration plate B1 and the second calibration plate B2 relative to the guide rail so that the distances between the first calibration plate B1 and the second calibration plate B2 and the depth camera are d1 and d2, respectively. Then, based on the principle that "all points on the same beam G have the same corresponding pixel on the image plane (i.e., M1 or M2) of the depth camera", the pixel coordinates (u,v) of the pixel point P on M1 are calculated using the intrinsic and extrinsic parameters of the depth camera. Finally, the first distance is calculated based on the pixel coordinates (u,v) of the pixel point P on M1.

[0038] In addition, since depth cameras have different types (i.e., i-TOF cameras and d-TOF cameras), when calculating the second distance based on the calculation principle of depth cameras, the different types of depth cameras should be fully considered, as the calculation methods for the second distance are different.

[0039] In one implementation, the calculation of the "second distance" in step 103 may include: obtaining the phase difference between the first intersection point and the second intersection point; and calculating the second distance based on the obtained phase difference. Specifically, in this implementation, the phase difference between the first intersection point A1 and the second intersection point A2 is substituted into a second formula to calculate the second distance; wherein, the second formula is expressed as:

[0040]

[0041] D′(P′) represents the second distance, and c represents the speed of light. This indicates the phase corresponding to the first intersection point A1. The phase corresponding to the second intersection point A2 is represented by π, and f is represented by f. m This indicates the modulation frequency of the i-TOF camera.

[0042] Different Time-of-Flight (TOF) cameras have different tap numbers and modulation methods, so the calculation process for the phase difference between the first intersection point A1 and the second intersection point A2 will also differ. However, for the coordinates (u′, v′) of the pixel P′ corresponding to the first intersection point A1 and the second intersection point A2 on the image plane M2 of the depth camera, the phase difference at the first intersection point A1 can be calculated based on the echo signals received by that pixel from the depth camera and reflected through the first intersection point A1 and the second intersection point A2. The phase corresponding to the second intersection point A2 This yields the phase difference between the first intersection point A1 and the second intersection point A2. Therefore, the second distance can be calculated based on the second formula mentioned above.

[0043] As another implementation, the calculation of the "second distance" in step 103 may include: obtaining the time difference between the first flight time and the second flight time; calculating the second distance based on the obtained time difference; wherein, the first flight time is the time elapsed from when the depth camera emits the same beam G until the depth camera receives the same beam G reflected back from the first intersection point A1 (i.e., the echo signal); the second flight time is the time elapsed from when the depth camera emits the same beam G until the depth camera receives the same beam G reflected back from the second intersection point A2. Specifically, in this implementation, the time difference between the first flight time and the second flight time is substituted into a third formula to calculate the second distance; wherein, the third formula is expressed as:

[0044]

[0045] Where D′(P′) represents the second distance, c represents the speed of light, t2 represents the second flight time, and t1 represents the first flight time.

[0046] In this embodiment, for the pixel P′ on the image plane M2 of the depth camera corresponding to the first intersection point A1 and the second intersection point A2, the first flight time t1 and the second flight time t2 are obtained based on the echo signal received from the coordinates (u′, v′) of the pixel P′, thereby obtaining the time difference Δt = t2 - t1 between the first flight time t1 and the second flight time t2, and then the second distance can be calculated according to the third formula mentioned above.

[0047] Furthermore, if the first distance and the second distance are the same, that is, D(P) and D′(P′) are the same, then it is determined that the image plane of the depth camera is parallel to the first calibration plate B1 and the second calibration plate B2, and thus it is determined that the depth camera is currently in an ideal state, and the camera coordinates of the first intersection point A1 and the second intersection point A2 on the image plane are the same, that is, (u,v)=(u′,v′); if the first distance and the second distance are different, then it is determined that the image plane of the depth camera is tilted relative to the first calibration plate B1 and the second calibration plate B2, that is, it is determined that the depth camera is currently in a non-ideal state, and the camera coordinates of the first intersection point A1 and the second intersection point A2 on the image plane are not the same, that is, (u,v)≠(u′,v′).

[0048] Therefore, as one implementation, if the image plane of the depth camera is tilted relative to the first calibration plate B1 and the second calibration plate B2 in step 105 (i.e., the depth camera is currently in a non-ideal state), then the depth camera needs to be calibrated to eliminate the adverse effects of the tilt error. In this implementation, calibrating the depth camera may include: obtaining the rotation matrix between the image plane of the depth camera in the non-ideal state and the image plane of the depth camera in the ideal state; and calibrating the depth camera according to the obtained rotation matrix.

[0049] Understandably, because the non-ideal state has a tilt error compared to the ideal state (i.e., the image plane M2 of the depth camera is tilted relative to the first calibration plate B1 and the second calibration plate B2), there is a rotation matrix between the image plane M2 of the depth camera in the non-ideal state and the image plane M1 of the depth camera in the ideal state. Specifically, the coordinates (u,v) of the pixel P corresponding to the first intersection point A1 and the second intersection point A2 on the image plane M1 of the depth camera in the ideal state satisfy the following relationship:

[0050]

[0051] Where z represents the distance between the guide rail and the first calibration plate B1 / second calibration plate B2, K represents the intrinsic parameter matrix of the depth camera, and q represents the coordinates of the pixel point P on the image plane M1 of the depth camera corresponding to the first intersection point A1 and the second intersection point A2 in the camera coordinate system.

[0052] Under non-ideal conditions, the coordinates (u′, v′) of the pixel P′ corresponding to the first intersection point A1 and the second intersection point A2 on the image plane M2 of the depth camera satisfy the following relationship:

[0053]

[0054] Where z represents the distance between the guide rail and the first calibration plate B1 / second calibration plate B2, K represents the intrinsic parameter matrix of the depth camera, q′ represents the coordinates of the pixel P′ corresponding to the first intersection point A1 and the second intersection point A2 on the image plane M2 of the depth camera in the camera coordinate system, and R represents the rotation matrix between the image plane M2 of the depth camera in the non-ideal state and the image plane M1 of the depth camera in the ideal state. Therefore, as long as the rotation matrix R between the image plane M2 of the depth camera in the non-ideal state and the image plane M1 of the depth camera in the ideal state is known, the correspondence between pixel P′ on image plane M2 and pixel P on image plane M1 can be easily established.

[0055] In one specific implementation, "obtaining the rotation matrix between the image plane of the depth camera in a non-ideal state and the image plane of the depth camera in an ideal state" in this embodiment may include: obtaining the initial rotation matrix between the image plane of the depth camera in a non-ideal state and the image plane of the depth camera in an ideal state according to the pose expression of the depth camera; the pose expression is as follows:

[0056]

[0057] The depth camera consists of n pixels, where n is a positive integer greater than 1. D(P,i) represents the first distance obtained from the i-th pixel in the depth camera, and D′(P′,i) represents the second distance obtained from the i-th pixel in the depth camera. i e represents the error between the first distance and the second distance obtained from the i-th pixel in the depth camera. i T e i The transpose of the matrix, R denotes the rotation matrix, and J denotes the Jacobian matrix operation. Let R be the minimum value of the objective function J(R) with rotation matrix R as a parameter, and let R be the initial rotation matrix when the objective function J(R) is minimized. After obtaining the initial rotation matrix, the depth camera can be calibrated and corrected to eliminate the adverse effects of tilt error.

[0058] In this specific implementation, since the rotation matrix between the image plane M2 of the depth camera in the non-ideal state and the image plane M1 of the depth camera in the ideal state is unknown, that is, the correspondence between the pixel P′ on the image plane M2 and the pixel P on the image plane M1 is unknown, the position of P′ can only be found based on the estimated value of the current extrinsic parameters of the depth camera. If the current extrinsic parameters of the depth camera are not ideal, the difference in distance between the pixel P′ on the image plane M2 and the pixel P on the image plane M1 will be large. Therefore, in order to reduce this difference, it is necessary to optimize the extrinsic parameters of the depth camera to find P′ that is more similar to P.

[0059] The criterion for determining whether a pixel P′ on image plane M2 is similar to a pixel P on image plane M1 is to calculate the error between the first distance and the second distance (i.e., the difference between the first distance and the second distance). The error between the first distance and the second distance is defined as e = D(P) - D′(P′). Considering that a depth camera includes multiple pixels (e.g., n pixels, where n is a positive integer greater than 1), the pose estimation problem of the depth camera becomes the pose expression described above. When minimized, R in the pose expression becomes the initial rotation matrix.

[0060] In another specific implementation, based on the previous implementation, the step of "obtaining the rotation matrix between the image plane of the depth camera in a non-ideal state and the image plane of the depth camera in an ideal state" in this embodiment may further include: calculating the corresponding pixel coordinates of the first and second intersection points in the depth camera based on the initial rotation matrix; calculating the third distance between the first and second intersection points based on the corresponding pixel coordinates of the first and second intersection points in the depth camera; differentiating the difference between the third distance and the first distance, and converting the result of the differentiation into a Jacobian matrix; and processing the Jacobian matrix by calculating the increment and iteratively solving using a nonlinear optimization algorithm to obtain the optimal rotation matrix. Furthermore, after obtaining the optimal rotation matrix, the depth camera can be calibrated and corrected using the obtained optimal rotation matrix to eliminate the adverse effects of tilt error. It is understood that since the optimal rotation matrix is ​​more effective than the initial rotation matrix in eliminating the adverse effects of tilt error, after obtaining the optimal rotation matrix, the depth camera is no longer calibrated and corrected using the initial rotation matrix, but rather using the optimal rotation matrix.

[0061] To clearly understand this specific implementation, it will be described in detail below:

[0062] Let p be the camera coordinates of the pixel P on the image plane M1 of the depth camera in the ideal state, corresponding to the first intersection point A1 and the second intersection point A2. Let q be the camera coordinates and s be the pixel coordinates of the pixel P′ on the image plane M2 of the depth camera in the non-ideal state, corresponding to the first intersection point A1 and the second intersection point A2. Then, the initial rotation matrix obtained from the previous specific implementation can be used to associate p, q, and s, as follows:

[0063]

[0064] Where z represents the distance between the guide rail and the first calibration plate B1 / second calibration plate B2, and K represents the intrinsic parameter matrix of the depth camera. It should be noted that s here is not the same as the pixel coordinates (u′, v′) of pixel point P′ on image plane M2 mentioned earlier. The s here is calculated using the initial rotation matrix in the previous specific implementation, that is, the pixel coordinates s after rotation are obtained here.

[0065] Next, the third distance between the first intersection point A1 and the second intersection point A2 is calculated using pixel coordinates s. This third distance is not identical to the pixel coordinates (u′, v′) of pixel point P′ on image plane M2 mentioned earlier. Although this third distance corresponds to the second distance mentioned earlier, it is calculated using the pixel coordinates obtained from the initial rotation matrix in the previous implementation and the echo signal received by the corresponding pixel in the depth camera receiving module. In other words, the third distance obtained here is actually the rotated second distance. After obtaining the third distance, the difference between the third distance and the first distance is calculated, i.e., the error e = D(p) - D′(s) is obtained; where D(p) represents the first distance and D′(s) represents the third distance. From e = D(p) - D′(s), we can see that e changes with D′(s), and D′(s) is related to the pixel coordinates s. The pixel coordinates s change with the initial rotation matrix R. Therefore, in order to further optimize the initial rotation matrix R, we need to minimize the error e between the third distance and the first distance. At this time, we take the derivative of the error e between the third distance and the first distance, and the expression is as follows:

[0066]

[0067] Where ξ is the Lie algebraic form of the initial rotation matrix R, and δξ ​​is the perturbation term.

[0068] The derivative expression of the error e between the third distance and the first distance is transformed into a Jacobian matrix, as shown below:

[0069]

[0070] in, Let be the gradient of the distance difference at pixel coordinate s.

[0071] Equivalently, in, Yes Differentiate,

[0072] This can be obtained by differentiating Lie algebras. Based on this, (Right now It can be represented by the following expression:

[0073]

[0074] Among them, f x with f y Let be the pixel size, and let q(X,Y) be the camera coordinates of pixel P′. Let z represent the distance between the guide rail and the first calibration plate B1 / second calibration plate B2. Further, let's connect this to... Combining these, we can obtain the Jacobian matrix, which is a more specific derivative expression of the error e between the third distance and the first distance.

[0075] After obtaining the Jacobian matrix, it can be processed using nonlinear optimization algorithms (such as the Gauss-Newton algorithm) to calculate increments and iteratively solve for the optimal rotation matrix. This optimal rotation matrix is ​​then used to calibrate and correct the depth camera. During the calibration and correction process using the optimal rotation matrix, the true distance between the first intersection point A1 and the second intersection point A2 can be calculated. Subsequent calibration procedures can follow the existing calibration procedures for i-TOF cameras, such as Wiggling and FPPN error calibration.

[0076] This embodiment provides a technique for calculating the true distance between the first intersection point A1 and the second intersection point A2 using a known rotation matrix (i.e., the initial rotation matrix or the optimal rotation matrix): In an ideal state, the corresponding camera coordinates of the corresponding pixels of the marker points (i.e., the first intersection point A1 and the second intersection point A2) on the depth camera image plane are known. Based on the initial rotation matrix or the optimal rotation matrix, the camera coordinates of the corresponding pixels of the marker points on the depth camera image plane after rotation can be obtained. Then, plane fitting is performed on the rotated marker plates (i.e., the first calibration plate B1 and the second calibration plate B2). The imaging of the marker points on the depth camera image plane satisfies perspective transformation, and the position information of the corresponding image plane can be obtained, that is, the position information of the image plane M2 of the depth camera in a non-ideal state can be obtained.

[0077] It should be understood that the above implementation methods are only preferred implementations of the embodiments of this application, and are not the only limitation on the specific process of step 105 in the embodiments of this application; those skilled in the art can make flexible settings based on the embodiments of this application and according to the actual application scenario.

[0078] Furthermore, this application embodiment also provides a calibration system, which includes a guide rail, a calibration plate slidably connected to the guide rail, a base, a depth camera, and a control and processor. The depth camera is placed on the base, and the base and the calibration plate are respectively set at both ends of the guide rail. The control and processor can control the calibration plate to slide on the guide rail, control the depth camera to emit light signals (i.e., the same light beam G mentioned above) to the calibration plate and receive echo signals reflected back from calibration plates at different distances (such as the first calibration plate B1 and the second calibration plate B2 mentioned above), and execute the above calibration method according to the received echo signals to complete the calibration of the depth camera.

[0079] In summary, the tilt problem of a depth camera can be viewed as the image plane of the depth camera rotating around its optical center at an angle. Once this rotation angle (equivalent to the initial or optimal rotation matrix) is calculated, the true distance between the first intersection point A1 and the second intersection point A2 can be determined. Experiments show that the embodiments of this application achieve good tilt correction for the depth camera; for details of the correction effect, please refer to [link to relevant documentation]. Figure 3 and Figure 4 ;in, Figure 3 This is a calibration effect diagram of the calibration method provided in the embodiments of this application at 1000mm. Figure 4 The calibration effect diagram of the calibration method provided in the embodiment of this application at 2000mm.

[0080] Figure 5 This is a block diagram of a depth camera provided in an embodiment of this application. This application also provides a depth camera, including a projection module 501, an acquisition module 502, a processing module 503, and a storage module 504. The storage module 504 stores calibration parameters (such as the initial rotation matrix or optimal rotation matrix mentioned above) obtained when executing the calibration method provided in this application. Specifically, the projection module 501 projects a light signal (i.e., the same light beam G mentioned above) onto the target area; the acquisition module 502 receives the echo signal reflected back from the target area; and the processing module 503 generates a depth image of the target area based on the reflected echo signal and corrects the depth image based on the calibration parameters in the storage module 504, thereby obtaining a corrected depth image.

[0081] Figure 6 This is a block diagram of a computer-readable storage medium provided in an embodiment of this application. An embodiment of this application also provides a computer-readable storage medium 600, on which executable instructions 610 are stored. When executed, the executable instructions 610 perform the calibration method provided in an embodiment of this application.

[0082] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0083] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk).

[0084] It should be noted that the various embodiments in this application are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For product-related embodiments, since they are similar to method-related embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method-related embodiments.

[0085] It should also be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0086] The above description of the disclosed embodiments enables those skilled in the art to implement or use the content of this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in this application may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A calibration method applied to a calibration system, the calibration system comprising a depth camera, a calibration board, and a control and processor, characterized in that, The method comprises: The depth camera's transmitting module is controlled to emit the same light beam to a first calibration plate and a second calibration plate that are at different distances and parallel to each other, and the depth camera's acquisition module is activated to receive the echo signals reflected back by each calibration plate; wherein the light beam has a first intersection point and a second intersection point with the first calibration plate and the second calibration plate, respectively; The camera coordinates of each intersection point are obtained, and the first distance between the first intersection point and the second intersection point of the depth camera under ideal conditions is calculated based on the camera coordinates and the preset intrinsic and extrinsic parameters of the depth camera; wherein, the ideal state indicates that the image plane of the depth camera is parallel to the calibration plate; The second distance between the first intersection point and the second intersection point is calculated based on the calculation principle of the depth camera and the echo signal. Based on the first distance and the second distance, it is determined whether the image plane of the depth camera is parallel to the calibration plate, and based on the determination result, it is selected whether the depth camera needs to be calibrated and corrected.

2. The calibration method as described in claim 1, characterized in that, The calculation method for the first distance includes: The first distance is calculated according to the first formula; wherein, the first formula is expressed as: D(P) represents the first distance, u represents the x-coordinate of the pixel corresponding to the first intersection point and the second intersection point on the image plane of the depth camera under ideal conditions, v represents the y-coordinate of the pixel corresponding to the first intersection point and the second intersection point on the image plane of the depth camera under ideal conditions, f represents the focal length of the depth camera, d1 represents the distance between the first calibration plate and the depth camera, d2 represents the distance between the second calibration plate and the depth camera, and c x c represents the x-coordinate of the center of the image plane of the depth camera under the ideal state. y This represents the ordinate of the center of the image plane of the depth camera under the ideal condition.

3. The calibration method as described in claim 1, characterized in that, The second distance is calculated in the following ways: The phase difference between the first intersection point and the second intersection point is obtained based on the echo signal; The second distance is calculated by substituting the phase difference into the second formula; wherein the second formula is expressed as: D′(P′) represents the second distance, and c represents the speed of light. This indicates the phase corresponding to the first intersection point. This represents the phase corresponding to the second intersection point, π represents pi, and f m This indicates the modulation frequency of the depth camera.

4. The calibration method as described in claim 1, characterized in that, The second distance is calculated in the following ways: Obtain the time difference between a first flight time and a second flight time; wherein, the first flight time is the time elapsed from the moment the depth camera emits the same beam until the depth camera receives the echo signal reflected back by the first calibration plate; the second flight time is the time elapsed from the moment the depth camera emits the same beam until the depth camera receives the echo signal reflected back by the second calibration plate. The second distance is calculated by substituting the time difference into the third formula; wherein the third formula is expressed as: D′(P′) represents the second distance, c represents the speed of light, t2 represents the second flight time, and t1 represents the first flight time.

5. The calibration method according to any one of claims 1-4, characterized in that, The step of determining whether the image plane of the depth camera is parallel to the calibration plate based on the first distance and the second distance includes: If the first distance is the same as the second distance, then it is determined that the image plane of the depth camera is parallel to the calibration plate, and the depth camera is in the ideal state; If the first distance is different from the second distance, it is determined that the image plane of the depth camera is tilted relative to the calibration plate, and the depth camera is in a non-ideal state; wherein, the non-ideal state indicates that the image plane of the depth camera is tilted relative to the calibration plate.

6. The calibration method as described in claim 5, characterized in that, The step of selecting whether to calibrate the depth camera based on the judgment result includes: If the depth camera is in the ideal state, the output will show a result that does not require calibration or correction of the depth camera. If the depth camera is in the non-ideal state, then the depth camera is calibrated and corrected.

7. The calibration method as described in claim 6, characterized in that, The calibration and correction of the depth camera includes: Obtain the rotation matrix between the image plane of the depth camera in the non-ideal state and the image plane of the depth camera in the ideal state; The depth camera is calibrated and corrected based on the rotation matrix.

8. The calibration method as described in claim 7, characterized in that, The step of obtaining the rotation matrix between the image plane of the depth camera in the non-ideal state and the image plane of the depth camera in the ideal state includes: Based on the pose expression of the depth camera, the initial rotation matrix between the image plane of the depth camera in the non-ideal state and the image plane of the depth camera in the ideal state is obtained; the pose expression is as follows: The depth camera comprises n pixels, where n is a positive integer greater than 1. D(P,i) represents the first distance obtained by the i-th pixel in the depth camera, and D′(P′,i) represents the second distance obtained by the i-th pixel in the depth camera. i e represents the error between the first distance and the second distance obtained from the i-th pixel in the depth camera. i T e i The transpose of the matrix, R denotes the rotation matrix, and J denotes the Jacobian matrix operation. Let R represent the minimum value of the objective function J(R) with rotation matrix R as a parameter, and when the objective function J(R) is minimized, R is the initial rotation matrix.

9. The calibration method as described in claim 8, characterized in that, The step of obtaining the rotation matrix between the image plane of the depth camera in the non-ideal state and the image plane of the depth camera in the ideal state further includes: Calculate the corresponding pixel coordinates of the first intersection point and the second intersection point in the depth camera based on the initial rotation matrix; The third distance between the first intersection point and the second intersection point is calculated based on the echo signal received by the corresponding pixel in the receiving module of the depth camera according to the pixel coordinates. The difference between the third distance and the first distance is differentiated, and the result is converted into a Jacobian matrix; The Jacobian matrix is ​​processed by a nonlinear optimization algorithm to calculate the increment and iteratively solve the problem, thus obtaining the optimal rotation matrix.

10. A calibration system, characterized in that, Includes a guide rail, a calibration plate slidably connected to the guide rail, a base, a depth camera, and a control and processor, wherein: The depth camera is placed on the base, and the base and the calibration plate are respectively set at both ends of the guide rail; The controller and processor are used to control the calibration plate to slide on the guide rail, control the depth camera to emit light signals to the calibration plate and brake the depth camera's acquisition module to receive echo signals reflected back from the calibration plate at different distances, and execute the method as described in any one of claims 1-9 according to the received echo signals to complete the calibration of the depth camera.

11. A depth camera, characterized in that, It includes a projection module, an acquisition module, a processing module, and a storage module, among which: The storage module is used to store calibration parameters obtained when performing the calibration method as described in any one of claims 1-9; The projection module is used to project light signals onto the target area; The acquisition module is used to receive the echo signal reflected back from the target area; The processing module is used to generate a depth image of the target area based on the reflected echo signal, and to correct the depth image based on the calibration parameters in the storage module to obtain a corrected depth image.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores executable instructions that, when executed, perform the method as described in any one of claims 1-9.

Images