ToF camera calibration method, device, electronic device and storage medium

Abstract

The present application discloses a calibration method, device, electronic device and storage medium for a ToF camera, which belongs to the field of computer vision technology. The calibration method includes: calculating the depth value of real-time Raw data collected by the ToF camera; compensating the depth value using a pre-established distance error model to obtain a calibration result; wherein the distance error model is obtained by fitting model parameters based on the measured depth values ​​of Raw data samples collected by the ToF camera at different measurement distances and the real depth values ​​of the Raw data samples at the different measurement distances, and the real depth value is obtained by curving the reference surface at the different measurement distances based on the intrinsic parameter value of the ToF camera.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and specifically relates to a calibration method, device, electronic device and storage medium for a ToF camera. Background Art

[0002] As a mainstream depth camera, Time-of-Flight (ToF) camera has become one of the best choices for obtaining distance depth maps of three-dimensional scenes due to its compact structure, fast response speed, simple algorithm, ability to provide three-dimensional images at high frame rate, and ability to provide intensity data and distance information for each pixel.

[0003] However, due to the existence of systematic errors and random errors, the measurement results and accuracy of the depth information of the unprocessed image (Raw) data collected by the ToF camera will be affected by many factors such as the internal and external environment of the camera system, which ultimately leads to low accuracy of the acquired depth information. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a calibration method, device, electronic device and storage medium for a ToF camera, which can solve the problem that the existing ToF camera has low measurement accuracy for depth information.

[0005] In a first aspect, an embodiment of the present application provides a calibration method for a ToF camera, the method comprising:

[0006] Calculate the depth value of the real-time Raw data collected by the ToF camera;

[0007] Using a pre-established distance error model, the depth value is compensated to obtain a calibration result;

[0008] Among them, the distance error model is obtained by fitting model parameters based on the measured depth values ​​of the Raw data samples collected by the ToF camera at different measurement distances and the true depth values ​​of the Raw data samples at the different measurement distances, and the true depth value is obtained by curving the reference surface at the different measurement distances based on the intrinsic parameter value of the ToF camera.

[0009] In a second aspect, an embodiment of the present application provides a calibration device for a ToF camera, the device comprising:

[0010] The first processing module is used to calculate the depth value of the real-time Raw data collected by the ToF camera;

[0011] Using a pre-established distance error model, the depth value is compensated to obtain a calibration result;

[0012] Among them, the distance error model is obtained by fitting model parameters based on the measured depth values ​​of the Raw data samples collected by the ToF camera at different measurement distances and the real depth values ​​of the Raw data measurement distances at the different measurement distances. The real depth value is obtained by curving the reference surface at the different measurement distances based on the intrinsic parameter value of the ToF camera.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0016] In an embodiment of the present application, by establishing a distance error model in advance based on the measured depth value measured by the ToF camera at the test position and the actual depth value of the position obtained by different methods, the distance error model can be used to perform depth compensation calibration on the Raw data collected by the ToF camera to eliminate errors when the ToF camera is actually used, thereby achieving higher measurement accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 One of the flow charts of the calibration method of the ToF camera provided in the embodiment of the present application;

[0018] Figure 2 The second flowchart of the calibration method of the ToF camera provided in the embodiment of the present application;

[0019] Figure 3 A schematic diagram of a process for constructing a distance error model in a ToF camera calibration method provided in an embodiment of the present application;

[0020] Figure 4 A schematic diagram of a process for performing internal parameter calibration on a ToF camera in a calibration method for a ToF camera provided in an embodiment of the present application;

[0021] Figure 5 A schematic diagram of an incomplete calibration plate image captured in a ToF camera calibration method provided according to an embodiment of the present application;

[0022] Figure 6 A schematic diagram of the structure of a calibration device for a ToF camera provided in an embodiment of the present application;

[0023] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0024] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0026] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0027] In order to solve the problem of low measurement accuracy in the prior art, the embodiment of the present application establishes a distance error model in advance based on the measured depth value measured by the ToF camera at the test position and the real depth value of the position obtained by different methods. When the ToF camera is actually used, the distance error model can be used to perform depth compensation calibration on the Raw data collected by the ToF camera to eliminate errors, thereby achieving higher measurement accuracy. The calibration method, device, electronic device and storage medium of the ToF camera provided in the embodiment of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0028] Figure 1 This is one of the flow charts of the calibration method of the ToF camera provided in the embodiment of the present application. The method can be executed by a terminal with a ToF function, such as Figure 1 As shown, the method includes:

[0029] Step 101, calculating the depth value of the real-time Raw data collected by the ToF camera.

[0030] It can be understood that when a ToF camera is actually used for data collection, the ToF camera can be used to collect real-time Raw data of the target three-dimensional scene, and then the depth value corresponding to the real-time Raw data can be calculated. For example, a four-step phase method can be used to calculate the depth value of the real-time Raw data.

[0031] Step 102: Compensate the depth value using a pre-established distance error model to obtain a calibration result.

[0032] Among them, the distance error model is obtained by fitting model parameters based on the measured depth values ​​of the Raw data samples collected by the ToF camera at different measurement distances and the true depth values ​​of the Raw data samples at the different measurement distances, and the true depth value is obtained by curving the reference surface at the different measurement distances based on the intrinsic parameter value of the ToF camera.

[0033] It can be understood that the embodiment of the present application can analyze the measurement error of the ToF camera through experiments in advance, and fit the distance error model of the ToF camera, so as to quickly and accurately obtain the distance depth map of the target scene when the ToF camera is used for data collection in the future.

[0034] Specifically, when fitting the distance error model of the ToF camera, the ToF camera can be used to collect the Raw data of the preset plane at multiple measurement distances of the preset plane as Raw data samples, and calculate the measured depth values ​​corresponding to the Raw data samples. At the same time, the reference surfaces at each measurement distance are determined respectively, and these reference surfaces are curved using the intrinsic parameter values ​​of the ToF camera to obtain the true depth values ​​of the Raw data samples at each measurement distance. Afterwards, the parameters of the distance error model are fitted by calculating the error between the measured depth value and the true depth value at each measurement distance, thereby determining the distance error model of the ToF camera.

[0035] On the basis of pre-establishing the distance error model of the ToF camera, the distance error model of the ToF camera can be used to quickly and accurately compensate the depth value calculated in step 101, and the depth value obtained after correction becomes the calibration result of the measurement result of the ToF camera.

[0036] That is to say, the embodiments of the present application can adopt different methods to obtain the real depth value of the collected data in advance, and establish a distance error model by measuring the depth value and the real depth value. Figure 2As shown, it is the second flow chart of the calibration method of the ToF camera provided in the embodiment of the present application. The calibration of the ToF camera in the embodiment of the present application refers to obtaining the correct depth value after correcting the distance error using the distance error model for the real-time Raw data collected by the ToF camera. Specifically, the depth value calculated by the four-step phase method of the real-time Raw data collected by the ToF camera is added with the distance error of the embodiment of the present application, and the distance measurement value of each pixel point of the image is compensated to obtain the calibrated depth value.

[0037] Among them, the ToF camera provides a distance depth map of a three-dimensional scene by using a low-cost complementary metal oxide semiconductor (CMOS) pixel array and active modulated light source technology. Each pixel in the resulting image can measure the brightness of the corresponding target and the arrival time of the reflected modulated light, thereby calculating the distance depth of the point.

[0038] RAW data is the original data that the CMOS or Charge Coupled Device (CCD) image sensor converts the captured light source signal into a digital signal. A RAW file is a file that records the original information of the digital camera sensor and some metadata (such as ISO settings, shutter speed, aperture value, white balance, etc.) generated by the camera. RAW is an unprocessed and uncompressed format.

[0039] Different measurement distances refer to the positions of the ToF camera at different distances from the above-mentioned preset plane. Since the ToF camera directly measures the distance to the preset plane at these positions, these positions are called different measurement distances. Correspondingly, the depth value directly measured by the ToF camera can be called the measured depth value, and the compensated and corrected depth value of the Raw data samples at different measurement distances can be called the real depth value.

[0040] The calibration method for a ToF camera provided in an embodiment of the present application establishes a distance error model in advance based on the measured depth value measured by the ToF camera at a test position and the actual depth value of the position obtained by different methods. When the ToF camera is actually used, the distance error model can be used to perform depth compensation calibration on the real-time Raw data collected by the ToF camera to eliminate errors, thereby achieving higher measurement accuracy.

[0041] Optionally, the calibration method of the ToF camera further includes a processing step of constructing a distance error model, such as Figure 3 FIG. 1 is a flow chart of constructing a distance error model in a ToF camera calibration method according to an embodiment of the present application, including:

[0042] Step 301 : Control the ToF camera to collect Raw data of the target white plane at the different measurement distances to form the Raw data samples, and calculate the measured depth values ​​of the Raw data samples at the different measurement distances.

[0043] It can be understood that when performing measurement error analysis of a ToF camera, the embodiment of the present application can first collect Raw data of the white plane at different distances from the white plane as Raw data samples, and then calculate the depth values ​​at different distances as measured depth values ​​according to algorithms such as the four-step phase method.

[0044] For example, the ToF camera can be fixed to a tripod, and the tripod can be fixed to a sliding rail to ensure that the relative position of the camera to the vertical white plane does not change. At the same time, the rangefinder is fixed to the sliding rail, and the distance between the rangefinder and the ToF camera and the photographed white plane is kept consistent to measure the position. The ToF camera uses a camera resolution of 180*240 and a fixed frequency of 20MHZ and 100MHZ, or other camera resolutions and frequencies.

[0045] Based on the parameter settings, the ToF camera can be used to collect Raw images at different distances at a fixed frequency. The measurement distance can be 600mm to 1000mm, with an interval of 100mm. The specific distance can be measured with a rangefinder to ensure that multiple measurements of the same distance are as consistent as possible. Different measurement distances are fixed by sliding guides, and a total of 5 frames are collected. The depth value of each frame is calculated by the four-step phase method as the measured depth value at different distances.

[0046] Step 302, respectively determining the reference surfaces at the different measurement distances, and using the intrinsic parameter value of the ToF camera to perform a curved surface transformation on the depth value of the reference surface, and obtaining the curved surface depth value as the true depth value of the Raw data sample at the different measurement distances;

[0047] It can be understood that in order to obtain the true depth value at different measurement distances relative to the white plane, different methods can be used to determine the reference surface at different measurement distances according to the position of the ToF camera. Afterwards, since the ToF camera adopts the principle of pinhole imaging, the measured depth value obtained in step 301 is a curved surface. According to the triangle similarity principle, based on the known intrinsic parameter value of the ToF camera and the depth value of the reference surface, the depth value of the reference surface can be curved, and the curved result is used as the true depth value at different measurement distances.

[0048] Optionally, respectively determining the reference plane at the different measuring distances includes: when the optical axis center of the ToF camera is perpendicular to the target white plane, using the plane at the different measuring distances as the reference plane; or, controlling the ToF camera to collect calibration Raw data of the target white plane at different calibration distances, and using a four-step phase method to respectively calculate the depth values ​​of the calibration Raw data at the different calibration distances; superimposing the depth difference from the different calibration distances to the different measuring distances on the depth value of the calibration Raw data, and using the superimposed depth value as the depth value of the reference plane; using the least squares method to fit the depth value of the reference plane into a plane, and using the fitted plane as the reference plane.

[0049] It can be understood that the embodiment of the present application can use the ToF camera to collect Raw data of the white plane at multiple calibration distances different from the measurement distance of step 301, and calculate the depth values ​​of the Raw data at these calibration distances according to the four-step phase method. In order to obtain the true depth value of the measured depth value in step 301, the depth value of the Raw data at these calibration distances is added\subtracted from the depth value of the difference between the calibration distance and the measurement distance to obtain a reference depth value, and the plane is fitted by the least squares method based on the reference depth value to obtain a reference surface. Then, the depth value of the entire reference surface obtained by fitting can be curved according to the intrinsic parameter value of the ToF camera to obtain the true depth value at different measurement distances.

[0050] Alternatively, before data collection, the position of the ToF camera can be adjusted so that the center of the optical axis of the ToF camera is perpendicular to the target white plane, and then the plane at each measuring distance where the ToF camera is located can be directly used as the reference plane.

[0051] For example, the ToF camera and the rangefinder can be fixed in the same manner as step 301, and the Raw data at different calibration distances can be collected in the same manner. It should be understood that in the embodiment of the present application, different calibration distances are multiple positions different from different measurement distances, which have a certain distance from the corresponding measurement distance, that is, there is a certain depth difference. The specific distance from the target white plane at different calibration distances can be 650mm to 1050mm, with an interval of 100mm, and a total of 5 frames are collected. The depth value of each frame is calculated by the four-step phase method.

[0052] In order to obtain the true depth value at 600mm to 1000mm in step 301, the depth value calculated for each frame is subtracted by 50mm as the depth value of the reference surface, and the depth value d of the reference surface is fitted to the plane ax+by+c=d by the least squares method as the reference surface at different measurement distances. On this basis, since the ToF camera adopts the pinhole imaging principle, the measurement value obtained in step 301 is a curved surface. According to the triangle similarity principle, if the camera internal parameters and the plane depth value d are known, the plane can be curved, and the result is used as the true depth value of the data collected by the ToF camera.

[0053] In addition, it can also be assumed that the center of the camera optical axis is perpendicular to the shooting plane, and the plane at the distance of 600mm to 1000mm measured in step 301 can be directly used as the absolute plane. On this basis, according to the camera pinhole imaging and the triangle similarity principle, when the camera internal parameters are known, the absolute plane can be curved, and the curved depth value can be used as the true depth value.

[0054] Among them, the formula for surfacing can be expressed as:

[0055]

[0056] Among them, d qu and d ping They represent the depth value after curvature and the depth value of the reference surface before curvature, respectively. f is the focal length of the camera. (i, j) is the coordinate of a certain pixel. x and y are the x-axis and y-axis coordinate variables, respectively. c is the intercept parameter of the reference surface plane.

[0057] The embodiment of the present application can effectively fit the measurement surface of the ToF camera by curving the depth value of the reference plane to obtain the depth value of the curved surface, compensate for the error introduced by pinhole imaging, and thus further improve the measurement accuracy.

[0058] Step 303: construct an initial distance error model based on the measured depth value and the true depth value, and fit the parameters of the initial distance error model by a least square method to obtain the distance error model.

[0059] It can be understood that the embodiment of the present application can construct an initial distance error model based on the measured depth values ​​and true depth values ​​at different measurement distances. Specifically, when the measured depth value and the true depth value are known, the distance error Δd is the measured depth value minus the true depth value. Assuming d is the measured depth value, the initial distance error model can be constructed including: Δd = p(1)*d 2 +p(2)*d+p(3), where p(1), p(2), and p(3) are three parameters to be fitted of the distance error model. In addition, optionally, the distance error model can also be expressed as a linear function.

[0060] Optionally, constructing the initial distance error model includes: acquiring an influence factor of temperature on a ToF sensor in the ToF camera, and constructing the initial distance error model based on the measured depth value, the true depth value, and the influence factor.

[0061] In other words, the influence of ambient temperature on the ToF sensor in the ToF camera can be converted into a mathematical quantity to constitute the temperature influence factor, and the influence factor is also added to the model, and the primary or secondary initial distance error model constructed above is modified to obtain the final initial distance error model. By adding the ambient temperature influence factor, the influence of ambient temperature on the measurement results can be effectively compensated, so that more reliable measurement results can be obtained under different ambient temperatures.

[0062] On the basis of constructing the initial distance error model, the quadratic initial distance error model or the linear initial distance error model can be fitted by the least square method to calculate the model parameters, thereby finally constructing a distance error model with known model parameters.

[0063] The embodiment of the present application obtains the measured depth values ​​of the Raw data collected by the ToF camera at different measurement distances, and obtains the corrected real depth values ​​at these positions, fits the measurement error of the ToF camera, and finally obtains the distance error model measured by the ToF camera. The measurement error of the ToF camera can be compensated quickly and accurately in subsequent applications, thereby effectively improving the measurement efficiency and measurement accuracy.

[0064] Optionally, the calibration method of the ToF camera also includes: controlling the ToF camera to capture multiple frames of calibration plate images under different camera postures, and based on the calibration plate images, using a camera calibration toolkit to perform internal parameter calibration on the ToF camera to obtain the internal parameter value of the ToF camera.

[0065] Among them, the multiple frames of calibration plate images are images generated by saving the confidence map converted from the calibration plate Raw data or images directly acquired by the ToF camera. The multiple frames of calibration plate images have the same size and are selected in the same order.

[0066] It can be understood that before constructing the distance error model of the ToF camera, the embodiment of the present application can perform internal parameter calibration on the ToF camera to obtain the internal parameter value of the ToF camera. Specifically, the calibration plate images can be collected at different postures of the ToF camera, and the existing calibration toolkit can be used to calibrate based on these calibration plate images. For example, these calibration plate images can be placed in the calibration toolkit directory, triggering the camera calibration process in the calibration toolkit, and the camera internal parameters can be obtained. When collecting the calibration plate image, the calibration plate image can be incomplete and can be freely selected according to the size of the calibration plate captured. This selection process must ensure that the calibration plate size selection of all images is consistent and the selection order is consistent (such as all clockwise or counterclockwise).

[0067] For example, the intrinsic calibration process of a ToF camera can be as follows: Figure 4 FIG. 1 is a schematic diagram of a process for performing internal calibration of a ToF camera in a calibration method for a ToF camera provided in an embodiment of the present application, which mainly includes the following processing flow:

[0068] First, attach the calibration plate to a vertical white plane, fix the ToF camera to a tripod, and rotate the tripod to obtain multiple frames of calibration plate images at different ToF camera positions. You can collect incomplete calibration plate images, such as Figure 5 The shape shown, Figure 5 A schematic diagram of an incomplete calibration plate image captured in the calibration method of a ToF camera according to an embodiment of the present application can collect 15-20 frames of calibration plate images of similar shapes. During the calibration plate image acquisition process, the distance from the ToF camera to the calibration plate can be within the range of the acquisition frequency, for example, a distance of 500mm to 800mm at a frequency of 20MHZ can be used, or it can be within the corresponding range at other frequencies such as 80MHZ, 100MHZ, 120MHZ, etc.

[0069] Then, use the Toolbox_calib toolkit that comes with Matlab for calibration. Convert the collected calibration plate Raw data into a confidence map and save it as a fixed-format image, such as jpg, bmp, etc. Alternatively, if you get a fixed-format image directly from the camera, you can use it directly. Put the obtained image in the directory of the calibration toolkit, set the size of the calibration plate and the length of the calibration plate grid through the Matlab command terminal, and then calibrate to get the camera internal parameters cx, cy, fx, and fy.

[0070] In one embodiment, Figure 5The size of the calibration plate shown is selected as a 7*6 rectangular area, the boundary points are selected in a counterclockwise direction, and the length of the calibration plate grid is 25mm. The embodiment of the present application can also select calibration plates of other sizes, as long as all calibration plate images can be selected into a rectangular area of ​​a fixed size, and the rectangular area selection direction of all calibration plate images is consistent. The length of the calibration plate grid can be, for example, 20mm to 50mm.

[0071] The method for calculating the confidence map in the embodiment of the present application can be as follows:

[0072] temp 1 =x 0 -x 2 temp 2 =x 3 -x 1 ;

[0073] confidence=abs(temp 1 )+abs(temp).

[0074] Among them, confidence represents confidence, x 0 、x 1 、x 2 、x 3 The optical signals representing the phases of 0°, 90°, 180° and 270° respectively can be directly read from the Raw data. abs means taking the absolute value.

[0075] The embodiment of the present application collects images of the calibration plate at different postures of the ToF camera and uses the existing calibration toolkit to perform internal parameter calibration operations on the ToF camera. The operation is simple and the response speed is fast.

[0076] It should be noted that the calibration method of the ToF camera provided in the embodiment of the present application can be executed by a calibration device of the ToF camera, or a control module in the calibration device of the ToF camera for executing the calibration method of the ToF camera. In the embodiment of the present application, the calibration method of the ToF camera performed by the calibration device of the ToF camera is taken as an example to illustrate the calibration device of the ToF camera provided in the embodiment of the present application.

[0077] The structure of the calibration device of the ToF camera in the embodiment of the present application is as follows: Figure 6 As shown, it is a structural schematic diagram of a calibration device for a ToF camera provided in an embodiment of the present application. The device can be used to implement the calibration process of the ToF camera in the above-mentioned calibration method embodiments of each ToF camera, and the device includes:

[0078] The first processing module 601 is used to calculate the depth value of the real-time Raw data collected by the ToF camera;

[0079] The second processing module is used to compensate the depth value by using a pre-established distance error model to obtain a calibration result.

[0080] Among them, the distance error model is obtained by fitting model parameters based on the measured depth values ​​of the Raw data samples collected by the ToF camera at different measurement distances and the true depth values ​​of the Raw data samples at the different measurement distances, and the true depth value is obtained by curving the reference surface at the different measurement distances based on the intrinsic parameter value of the ToF camera.

[0081] Optionally, the calibration device of the ToF camera further includes:

[0082] a third processing module, configured to control the ToF camera to collect the Raw data of the target white plane at the different measurement distances to form the Raw data samples, and to calculate the measured depth values ​​of the Raw data samples at the different measurement distances;

[0083] A fourth processing module is used to respectively determine the reference surface at the different measurement distances, and use the intrinsic parameter value of the ToF camera to perform a curved surface transformation on the depth value of the reference surface to obtain the curved surface depth value as the true depth value of the Raw data sample at the different measurement distances;

[0084] The fifth processing module is used to construct an initial distance error model based on the measured depth value and the true depth value, and to fit the parameters of the initial distance error model by a least square method to obtain the distance error model.

[0085] Optionally, the fourth processing module, when used for respectively determining the reference surfaces at the different measurement distances, is used to:

[0086] Controlling the ToF camera to collect calibration Raw data of the target white plane at different calibration distances, respectively, and using the four-step phase method to calculate the depth values ​​of the calibration Raw data at the different calibration distances, respectively; superimposing the depth difference from the different calibration distances to the different measurement distances on the depth value of the calibration Raw data, and using the superimposed depth value as the depth value of the reference surface; using the least squares method to fit the depth value of the reference surface into a plane, and using the fitted plane as the reference surface;

[0087] or,

[0088] When the optical axis center of the ToF camera is perpendicular to the target white plane, the planes at different measurement distances are used as the reference planes.

[0089] Optionally, the fifth processing module, when used for constructing the initial distance error model, is used to:

[0090] The influence factor of temperature on the ToF sensor in the ToF camera is obtained, and the initial distance error model is constructed based on the measured depth value, the true depth value and the influence factor.

[0091] Optionally, the calibration device of the ToF camera further includes:

[0092] A sixth processing module, used to control the ToF camera to collect multiple frames of calibration plate images under different camera postures, and based on the calibration plate images, use a camera calibration toolkit to perform internal parameter calibration on the ToF camera to obtain the internal parameter value of the ToF camera;

[0093] Among them, the multiple frames of calibration plate images are images generated by saving the confidence map converted from the calibration plate Raw data or images directly acquired by the ToF camera. The multiple frames of calibration plate images have the same size and are selected in the same order.

[0094] The calibration device of the ToF camera in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0095] The calibration device of the ToF camera in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0096] The calibration device of the ToF camera provided in the embodiment of the present application can achieve Figures 1 to 5 The various processes implemented by the method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described here.

[0097] Alternatively, if Figure 7 As shown, the embodiment of the present application also provides an electronic device 700, including a processor 701, a memory 702, and a program or instruction stored in the memory 702 and executable on the processor 701. When the program or instruction is executed by the processor 701, each process of the calibration method embodiment of the above-mentioned ToF camera is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0098] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0099] Figure 8 The hardware structure diagram of an electronic device implementing the embodiment of the present application is shown in FIG. Figure 8 As shown, the electronic device 800 includes but is not limited to: a radio frequency unit 801, a network module 802, an audio output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a memory 809, and a processor 810 and other components.

[0100] Those skilled in the art will appreciate that the electronic device 800 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 810 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0101] The processor 810 is used to calculate the depth value of the real-time Raw data collected by the ToF camera, and to compensate the depth value using a pre-established distance error model to obtain a calibration result;

[0102] Among them, the distance error model is obtained by fitting model parameters based on the measured depth values ​​of the Raw data samples collected by the ToF camera at different measurement distances and the true depth values ​​of the Raw data samples at the different measurement distances, and the true depth value is obtained by curving the reference surface at the different measurement distances based on the intrinsic parameter value of the ToF camera.

[0103] In an embodiment of the present application, by establishing a distance error model in advance based on the measured depth value measured by the ToF camera at the test position and the actual depth value of the position obtained by different methods, the distance error model can be used to perform depth compensation calibration on the Raw data collected by the ToF camera to eliminate errors when the ToF camera is actually used, thereby achieving higher measurement accuracy.

[0104] Optionally, the input unit 804 in the embodiment of the present application is a ToF camera, which is used to collect Raw data of the target white plane at the different measurement distances to form the Raw data samples;

[0105] The processor 810 is also used to respectively calculate the measured depth values ​​of the Raw data samples at the different measurement distances; and, respectively determine the reference surfaces at the different measurement distances, and use the intrinsic parameter value of the ToF camera to perform a curved surface transformation on the depth value of the reference surface to obtain the curved surface depth value as the true depth value of the Raw data samples at the different measurement distances; and, based on the measured depth value and the true depth value, construct an initial distance error model, and fit the parameters of the initial distance error model by the least squares method to obtain the distance error model.

[0106] The embodiment of the present application obtains the measured depth values ​​of the Raw data collected by the ToF camera at different measurement distances, and obtains the corrected real depth values ​​at these positions, fits the measurement error of the ToF camera, and finally obtains the distance error model measured by the ToF camera. The measurement error of the ToF camera can be compensated quickly and accurately in subsequent applications, thereby effectively improving the measurement efficiency and measurement accuracy.

[0107] Optionally, the input unit 804 in the embodiment of the present application is a ToF camera, which is used to collect calibration Raw data of the target white plane at different calibration distances respectively;

[0108] The processor 810 is further configured to respectively calculate the depth values ​​of the calibration Raw data at the different calibration distances by using a four-step phase method; superimpose the depth difference from the different calibration distances to the different measurement distances on the depth value of the calibration Raw data, and use the superimposed depth value as the depth value of the reference surface; and fit the depth value of the reference surface into a plane by using a least squares method, and use the fitted plane as the reference surface;

[0109] or,

[0110] It is used to use the planes at different measuring distances as the reference planes when the optical axis center of the ToF camera is perpendicular to the target white plane.

[0111] The embodiment of the present application can effectively fit the measurement surface of the ToF camera by curving the depth value of the reference plane to obtain the depth value of the curved surface, compensate for the error introduced by pinhole imaging, and thus further improve the measurement accuracy.

[0112] Optionally, the processor 810 is further configured to obtain an influence factor of temperature on a ToF sensor in the ToF camera, and construct the initial distance error model based on the measured depth value, the true depth value, and the influence factor.

[0113] The embodiment of the present application can effectively compensate for the influence of the ambient temperature on the measurement result by adding the ambient temperature influence factor, so that more reliable measurement results can be obtained under different ambient temperatures.

[0114] Optionally, the input unit 804 in the embodiment of the present application is a ToF camera, which is used to collect multiple frames of calibration plate images under different camera postures;

[0115] The processor 810 is further configured to perform an internal parameter calibration on the ToF camera based on the calibration plate image using a camera calibration toolkit to obtain the internal parameter value of the ToF camera;

[0116] Among them, the multiple frames of calibration plate images are images generated by saving the confidence map converted from the calibration plate Raw data or images directly acquired by the ToF camera. The multiple frames of calibration plate images have the same size and are selected in the same order.

[0117] The embodiment of the present application collects images of the calibration plate at different postures of the ToF camera and uses the existing calibration toolkit to perform internal parameter calibration operations on the ToF camera. The operation is simple and the response speed is fast.

[0118] It should be understood that in the embodiment of the present application, the input unit 804 may include a graphics processor (Graphics Processing Unit, GPU) 8041 and a microphone 8042, and the graphics processor 8041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 806 may include a display panel 8061, and the display panel 8061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 807 includes a touch panel 8071 and other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 may include two parts: a touch detection device and a touch controller. Other input devices 8072 may include but are not limited to a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 809 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 810 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface, and the application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 810.

[0119] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the calibration method embodiment of the above-mentioned ToF camera is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0120] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0121] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned ToF camera calibration method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0122] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0123] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0125] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A calibration method for a ToF camera, characterized in that: The method comprises: Calculate the depth value of the real-time Raw data collected by the ToF camera; Using a pre-established distance error model, the depth value is compensated to obtain a calibration result; Among them, the distance error model is obtained by fitting model parameters based on the measured depth values ​​of the Raw data samples collected by the ToF camera at different measurement distances and the real depth values ​​of the Raw data samples at the different measurement distances, and the real depth value is obtained by curving the reference surface at the different measurement distances based on the internal parameter value of the ToF camera; the determination method of the reference surface at the different measurement distances includes: controlling the ToF camera to collect calibration Raw data of the target white plane at different calibration distances, and using the four-step phase method to calculate the depth values ​​of the calibration Raw data at the different calibration distances; superimposing the depth difference from the different calibration distances to the different measurement distances on the depth value of the calibration Raw data, and using the superimposed depth value as the depth value of the reference surface; using the least squares method to fit the depth value of the reference surface into a plane, and using the fitted plane as the reference surface; or, when the optical axis center of the ToF camera is perpendicular to the target white plane, using the planes at different measurement distances as the reference surface.

2. The calibration method of a ToF camera according to claim 1, characterized in that: Also includes: Controlling the ToF camera to collect Raw data of the target white plane at the different measurement distances to form the Raw data samples, and respectively calculating the measured depth values ​​of the Raw data samples at the different measurement distances; Determine the reference surfaces at the different measurement distances respectively, and use the intrinsic parameter value of the ToF camera to perform a curved surface transformation on the depth value of the reference surface, and obtain the curved surface depth value as the true depth value of the Raw data sample at the different measurement distances; An initial distance error model is constructed based on the measured depth value and the true depth value, and the parameters of the initial distance error model are fitted by a least square method to obtain the distance error model.

3. The calibration method of a ToF camera according to claim 2, characterized in that: The constructing of the initial distance error model comprises: The influence factor of temperature on the ToF sensor in the ToF camera is obtained, and the initial distance error model is constructed based on the measured depth value, the true depth value and the influence factor.

4. The calibration method of a ToF camera according to claim 1 or 2, characterized in that: Also includes: Control the ToF camera to collect multiple frames of calibration plate images under different camera postures, and based on the calibration plate images, use a camera calibration toolkit to perform internal parameter calibration on the ToF camera to obtain the internal parameter value of the ToF camera; Among them, the multiple frames of calibration plate images are images generated by saving the confidence map converted from the calibration plate Raw data or images directly acquired by the ToF camera. The multiple frames of calibration plate images have the same size and the selection order of the multiple frames of calibration plate images is consistent.

5. A calibration device for a ToF camera, characterized in that: The device comprises: The first processing module is used to calculate the depth value of the real-time Raw data collected by the ToF camera; A second processing module is used to compensate the depth value by using a pre-established distance error model to obtain a calibration result; The distance error model is obtained by fitting model parameters based on the measured depth values ​​of the Raw data samples collected by the ToF camera at different measurement distances and the real depth values ​​of the Raw data samples at the different measurement distances, and the real depth value is obtained by curving the reference surface at the different measurement distances based on the intrinsic parameter value of the ToF camera; It also includes: a fourth processing module, when determining the reference plane at the different measurement distances respectively, used to: control the ToF camera to collect calibration Raw data of the target white plane at different calibration distances respectively, and use the four-step phase method to calculate the depth values ​​of the calibration Raw data at the different calibration distances respectively; superimpose the depth difference from the different calibration distances to the different measurement distances on the depth value of the calibration Raw data, and use the superimposed depth value as the depth value of the reference plane; use the least squares method to fit the depth value of the reference plane into a plane, and use the fitted plane as the reference plane; or, when the optical axis center of the ToF camera is perpendicular to the target white plane, use the plane at the different measurement distances as the reference plane.

6. The calibration device for a ToF camera according to claim 5, characterized in that: Also includes: a third processing module, configured to control the ToF camera to collect the Raw data of the target white plane at the different measurement distances to form the Raw data samples, and to calculate the measured depth values ​​of the Raw data samples at the different measurement distances; The fourth processing module is used to respectively determine the reference surface at the different measurement distances, and use the intrinsic parameter value of the ToF camera to perform a curved surface transformation on the depth value of the reference surface to obtain the curved surface depth value as the true depth value of the Raw data sample at the different measurement distances; The fifth processing module is used to construct an initial distance error model based on the measured depth value and the true depth value, and to fit the parameters of the initial distance error model by a least square method to obtain the distance error model.

7. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the ToF camera calibration method according to any one of claims 1 to 4.

8. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the ToF camera calibration method according to any one of claims 1 to 4 are implemented.

Images