Parameter calibration method, depth image generation method and related components

Abstract

The application is suitable for the technical field of sensors, and provides a parameter calibration method, a depth image generation method and related components. The parameter calibration method of the time-of-flight image sensor comprises the following steps: determining a calibration step length according to a linear interval satisfied by reflected light intensity, wherein the linear interval is obtained by dividing the range of a linear quantization function corresponding to an arctangent function; determining a calibration point position for shooting a calibration image according to the calibration step length; obtaining calibration images of each phase shot at the calibration point position; for each pixel point in the image sensor, fitting the pixel values of the calibration images of the corresponding phase at different distances to obtain a fitting curve; the fitting curve represents the corresponding relationship between the distance and the pixel value, and the fitting curve is used as a sensor parameter for generating a depth image. The embodiments of the application can improve the reliability of depth information.

Description

Technical Field

[0001] This application belongs to the field of sensor technology, and in particular relates to a parameter calibration method, a depth image generation method, and related components. Background Technology

[0002] A Time-of-Flight (ToF) sensor is a device that uses light, sound waves, or other forms of beams to measure the distance between an object and a sensor. It calculates distance by sending a beam (such as a light pulse) and recording the time it takes for the beam to travel from transmission to reception. By measuring this time, ToF sensors can accurately determine the distance between an object and the sensor, making them widely used in various applications such as robot navigation, autonomous vehicles, industrial automation, and consumer electronics.

[0003] The time-of-flight sensor involved in this application is an image sensor. A light source synchronized with the time-of-flight image sensor emits infrared light of a specific wavelength. After the infrared light illuminates an object, the time-of-flight image sensor calculates depth information and generates a depth image by using the reflected infrared light.

[0004] However, research has found that the depth images obtained by these technologies deviate significantly from the true depth images. A solution is needed to reduce the error in depth measurement by time-of-flight image sensors. Summary of the Invention

[0005] This application provides a parameter calibration method for a time-of-flight image sensor, a method for generating depth images, and related components, which can improve the reliability of depth information.

[0006] The first aspect of this application provides a parameter calibration method for a time-of-flight image sensor, comprising: determining calibration points for capturing calibration images based on a linear interval satisfied by the intensity of reflected light, wherein the linear interval is obtained by dividing the range of a linear quantization function corresponding to an arctangent function; acquiring calibration images of each phase captured at the calibration points; for each pixel in the time-of-flight image sensor, fitting a fitting curve based on the pixel values ​​of the calibration images of the corresponding phases at different distances; wherein the fitting curve characterizes the correspondence between distance and pixel values, and the fitting curve is used as sensor parameters for generating depth images.

[0007] A second aspect of this application provides a method for generating a depth image, comprising: acquiring sensor parameters obtained by calibrating a time-of-flight image sensor, the sensor parameters being obtained according to the parameter calibration method for a time-of-flight image sensor described in the first aspect; determining pixel value information for each pixel based on the sensor parameters; and generating a first depth image based on the pixel value information.

[0008] A parameter calibration device for a time-of-flight image sensor, provided in a third aspect of this application, includes: a point determination unit, configured to determine calibration points for capturing calibration images based on a linear interval satisfied by the intensity of reflected light, wherein the linear interval is obtained by dividing the range of a linear quantization function corresponding to the arctangent function; an image acquisition unit, configured to acquire calibration images of each phase captured at the calibration points; and a calibration unit, configured to fit a fitting curve for each pixel in the time-of-flight image sensor based on the pixel values ​​of the corresponding phase calibration images at different distances; wherein the fitting curve characterizes the correspondence between distance and pixel values, and the fitting curve is used as sensor parameters for generating depth images.

[0009] A depth image generation apparatus provided in the fourth aspect of this application includes: a parameter acquisition unit for acquiring sensor parameters obtained by calibrating a time-of-flight image sensor, wherein the sensor parameters are obtained according to the parameter calibration method of the time-of-flight image sensor described in the first aspect; a pixel value determination unit for determining pixel value information of each pixel point according to the sensor parameters; and an image generation unit for generating a first depth image according to the pixel value information.

[0010] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the parameter calibration method for the time-of-flight image sensor described above, or, when executed by a processor, implements the steps of the depth image generation method described above.

[0011] A sixth aspect of this application provides a calibration device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the parameter calibration method for the time-of-flight image sensor described above.

[0012] A seventh aspect of this application provides a time-of-flight image sensor, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the depth image generation method described above.

[0013] An eighth aspect of this application provides a time-of-flight image sensor system, comprising: a light source, a time-of-flight image sensor, a light source driver chip, and a lens, wherein the time-of-flight image sensor is used to perform the steps of the depth image generation method described above.

[0014] A ninth aspect of this application provides a computer program product that, when run on a calibration device, causes the calibration device to execute the parameter calibration method of the time-of-flight image sensor; and when run on a time-of-flight image sensor, causes the time-of-flight image sensor to execute the depth image generation method.

[0015] In the embodiments of this application, a linear interval is obtained by dividing the range of the linear quantization function corresponding to the arctangent function. Based on the linear interval satisfied by the reflected light intensity, the calibration point for capturing calibration images is determined. Calibration images of each phase captured at the calibration point are obtained. The pixel values ​​of the calibration images of the corresponding phases at different distances are fitted to obtain a fitting curve. When generating a depth image using this fitting curve, since the calibration point is selected with reference to the linear interval, it is beneficial to reduce the inconsistency of the maximum error in different segments and reduce the problem of discontinuity in depth value calculation caused by the arctangent function being a nonlinear function. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram illustrating the implementation process of a parameter calibration method for a time-of-flight image sensor provided in an embodiment of this application;

[0018] Figure 2 This is a schematic diagram illustrating the implementation process of temperature calibration provided in the embodiments of this application;

[0019] Figure 3 This is a schematic diagram illustrating the implementation process of light source calibration provided in the embodiments of this application;

[0020] Figure 4 This is a schematic diagram illustrating the implementation process of a depth image generation method provided in an embodiment of this application;

[0021] Figure 5 This is a schematic diagram illustrating the specific implementation process of generating a second depth image provided in an embodiment of this application;

[0022] Figure 6 This is a schematic diagram of image segmentation provided in an embodiment of this application;

[0023] Figure 7 This is a schematic diagram of the structure of a parameter calibration device for a time-of-flight image sensor provided in an embodiment of this application;

[0024] Figure 8 This is a schematic diagram of the structure of a depth image generation device provided in an embodiment of this application;

[0025] Figure 9 This is a schematic diagram of the calibration device provided in the embodiments of this application;

[0026] Figure 10 This is a schematic diagram of the structure of the time-of-flight image sensor provided in the embodiments of this application;

[0027] Figure 11 This is a schematic diagram of the time-of-flight image sensor system provided in the embodiments of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are protected by this application.

[0029] A time-of-flight (ToF) sensor is a device that uses light, sound waves, or other forms of beams to measure the distance between an object and a sensor. It calculates distance by sending a beam (such as a light pulse) and recording the time it takes for the beam to travel from transmission to reception. By measuring this time, ToF sensors can accurately determine the distance between an object and the sensor, making them widely used in various applications such as robot navigation, autonomous vehicles, industrial automation, and consumer electronics.

[0030] The time-of-flight sensor involved in this application is an image sensor. A light source synchronized with the time-of-flight image sensor emits infrared light of a specific wavelength. After the infrared light illuminates an object, the time-of-flight image sensor calculates depth information by using the reflected infrared light.

[0031] However, research has found that the depth images obtained by these technologies deviate significantly from the actual depth images.

[0032] Specifically, the typical distance calculation method for time-of-flight image sensors is as follows:

[0033]

[0034] Where d is the distance, c is the speed of light, and t is the round-trip time of infrared light.

[0035] However, the round-trip time of infrared light is usually difficult to measure. Therefore, in time-of-flight image sensor solutions based on image sensors, the phase difference between the round-trip infrared light is typically measured. The distance is calculated using this method as follows:

[0036]

[0037] Where d is the distance and λ is the wavelength of the transmitted signal. It is the phase difference between the transmitted signal and the received signal.

[0038] Since image sensors can generally only obtain intensity information of reflected light and cannot directly obtain phase information of reflected light, time-of-flight image sensors can modulate the light source, send infrared light at a specific frequency, sample the reflected light at a specific phase in each light source cycle according to the frequency, and calculate the distance based on the results of these samplings.

[0039] If the sampling frequency is 4 times, sampling is typically performed at phases of 0°, 90°, 180°, and 270°. The calculation formula in this case is:

[0040]

[0041] Where f is the modulation frequency of the light source, c is the speed of light, A0 is the intensity of reflected light at 0° phase, A1 is the intensity of reflected light at 90° phase, A2 is the intensity of reflected light at 180° phase, and A3 is the intensity of reflected light at 270° phase. Thus, the depth information corresponding to each pixel can be calculated using the pixel values ​​of the time-of-flight image sensor to generate a depth image.

[0042] However, the above formula only applies to the most ideal situation. If the depth value is calculated directly using the formula, the resulting depth image will deviate significantly from the true depth image. On the one hand, to allow users to directly obtain a depth image, the tan... -1 The function is implemented in the hardware circuit through piecewise linear fitting with a fixed bit width, while tan -1 It is a non-linear function, and the maximum error is inconsistent in different segments, which will cause discontinuity in depth value calculation. On the other hand, time-of-flight image sensors have special fixed pattern noise (FPN). This FPN is different from that of ordinary image sensors. The FPN of ordinary image sensors is only related to factors such as exposure time, temperature, and pixel size, and only needs to be calibrated once at the factory. However, the FPN of time-of-flight image sensors is related to factors such as light source, reflectivity of the object being photographed, and distance between the object being photographed and the time-of-flight image sensor, in addition to the above factors.

[0043] To address the aforementioned causes of deviation, this application proposes a parameter calibration method for a time-of-flight image sensor and a depth image generation method, which can reduce the error in depth measurement by the time-of-flight image sensor.

[0044] To illustrate the technical solution of this application, specific embodiments are described below.

[0045] Figure 1 The illustration shows a schematic flowchart of a parameter calibration method for a time-of-flight image sensor according to an embodiment of this application. This method can be applied to a calibration device. The calibration device can be used to calibrate a time-of-flight image sensor.

[0046] Specifically, the parameter calibration method for the aforementioned time-of-flight image sensor may include the following steps S101 to S103.

[0047] Step S101: Determine the calibration point for capturing the calibration image based on the linear range satisfied by the reflected light intensity.

[0048] The linear interval is obtained by dividing the range of the linear quantization function corresponding to the arctangent function.

[0049] In the embodiments of this application, the linear quantization function refers to a function obtained by dividing the arctangent function into segments of a specific width and performing linear fitting on each segment. The range of the linear quantization function can be [0,∞], and dividing this range can yield several linear intervals. The interval lengths of different linear intervals can be the same or different.

[0050] In some embodiments of this application, tan -1 The linear quantization function of (x) is as follows:

[0051]

[0052] Where [0,0.5], [0.5,1], [1,2], [2,4] and [4,∞] each represent a linear interval.

[0053] In the embodiments of this application, a basic step size can be determined based on the range of the time-of-flight image sensor, an original point position can be determined based on the basic step size, and points can be added based on the original point position according to the linear interval satisfied by the reflected light intensity to obtain the calibration point position.

[0054] Specifically, the base step size for calibration can be determined based on the measurement range of the time-of-flight image sensor, following the principle of equal division. For example, when the system's measurement range is 1 to 5 meters, 41 initial points can be determined using a step size of 0.1 meters.

[0055] In the linear region where the quantization circuit has poor performance, the number of points can be increased. Specifically, this can be calculated based on the intensity of reflected light. It can be determined The linear interval falls into which the corresponding calibration point is added.

[0056] The calibration points corresponding to each linear interval can be set according to the actual situation.

[0057] In some embodiments of this application, the number of points can be increased by accumulating the step size on the exponent. For example, if calibration is needed in the interval [2,4], and 5 points need to be calibrated in this interval, then the step size is:

[0058]

[0059] At this point, the number of points added is 2. 1 =2,2 1+0.25 ≈2.38, 2 1+0.25*2 ≈2.83, 2 1+0.25*3 ≈3.36, 2 2 =4 of these 5 points.

[0060] Step S102: Obtain calibration images of each phase captured at the calibration point.

[0061] In the embodiments of this application, after the calibration points are determined, the time-of-flight image sensor can capture images at each calibration point. At each calibration point, multiple phases can be captured. For example, for each calibration point, sampling can be performed once at phases of 0°, 90°, 180°, and 270°. Of course, the number of phases at each calibration point, and the selected phases, can be set according to actual conditions.

[0062] It should be noted that during parameter calibration, the subject is a diffuse reflective plane that is completely parallel to the time-of-flight image sensor, such as a flat white wall.

[0063] To improve calibration accuracy, in some embodiments of this application, multiple calibration images are generated for each phase at each calibration point. After acquiring the calibration images of each phase taken at the calibration point, the parameter calibration method further includes: removing abnormal images from the calibration images; for each phase at the calibration point, averaging the remaining calibration images after removing abnormal images to obtain an average image. The average image can be used for subsequent calibration processing.

[0064] Furthermore, during the shooting process, the operating temperature of the time-of-flight image sensor can be kept constant, as well as the exposure time and gain remain constant.

[0065] Step S103: For each pixel in the time-of-flight image sensor, fit the pixel value of the corresponding phase calibration image at different distances to obtain the fitting curve.

[0066] The fitted curve represents the correspondence between distance and pixel value. This fitted curve can be used as a sensor parameter for generating depth images to achieve distance error correction. Any existing curve fitting algorithm can be used for fitting.

[0067] Specifically, for each pixel in the time-of-flight image sensor, a cubic spline function can be used to fit the pixel value at the corresponding phase and pixel position at different distances, so that each pixel can obtain a fitting curve at each phase.

[0068] In some embodiments of this application, pixel p is assumed to be at a distance d i The pixel value at that location is S p (d i If the fitted curve S is obtained, then... p (d) is:

[0069]

[0070] Among them, a k The fitting coefficients are denoted as .

[0071] The obtained fitted curve can be input into the component that performs the distance error correction process, namely the processing circuit inside the aforementioned time-of-flight image sensor or other processor chip, to eliminate distance measurement errors introduced by nonlinear functions, light sources, lenses, etc.

[0072] In the embodiments of this application, a linear interval is obtained by dividing the range of the linear quantization function corresponding to the arctangent function. Based on the linear interval satisfied by the reflected light intensity, the calibration point for capturing calibration images is determined. Calibration images of each phase captured at the calibration point are obtained. The pixel values ​​of the calibration images of the corresponding phases at different distances are fitted to obtain a fitting curve. When generating a depth image using this fitting curve, since the calibration point is selected with reference to the linear interval, it is beneficial to reduce the inconsistency of the maximum error in different segments and reduce the problem of discontinuity in depth value calculation caused by the arctangent function being a nonlinear function.

[0073] In some embodiments of this application, after obtaining the fitted curve, the parameter calibration method may further include: segmenting the fitted curve according to distance intervals to obtain a piecewise linear function, wherein the linear function corresponding to different distance intervals in the piecewise linear function is different.

[0074] Specifically, since the calibration process needs to be completed inside the time-of-flight image sensor or other processor chip, the fitted curve is fitted again with a piecewise linear function to reduce power consumption. In some embodiments of this application, the piecewise linear function is denoted as L. p (d), then we have:

[0075]

[0076] Where m and c are the fitting coefficients, and N is the total number of segments. The resulting fitted curve can be input into the terminal that performs the distance error correction process to eliminate distance measurement errors introduced by nonlinear functions and factors such as light sources and lenses.

[0077] In some embodiments of this application, the time-of-flight image sensor can also be calibrated by temperature. Specifically, the calibration image can include images captured at preset distances at multiple operating temperatures of the time-of-flight image sensor. In this case, such as... Figure 2 As shown, the parameter calibration method may also include steps S201 to S203.

[0078] Step S201: Determine the first phase corresponding to the preset distance.

[0079] Step S202: Determine the second phase based on the images corresponding to multiple operating temperatures and the linear quantization function.

[0080] Step S203: Determine the temperature drift correction function based on the first phase, the second phase, and multiple operating temperatures.

[0081] Among them, the temperature drift correction function can be used as a sensor parameter to correct the phase value.

[0082] The aforementioned operating temperatures can be set according to actual conditions, for example, by dividing the normal operating temperature range of the time-of-flight image sensor into specified steps.

[0083] Specifically, considering the distance error caused by temperature, time-of-flight image sensors (TOF image sensors) often generate more heat than traditional image sensor systems due to the presence of a light source. Therefore, the pixel values ​​applied to TOF image sensors will change significantly with temperature. Since the temperature-induced pixel value change is independent of the distance to the object being photographed and is relatively linear, it can be fitted using a linear function. This calibration process can maintain the same exposure time at a fixed distance while varying the operating temperature of the TOF image sensor. Assuming the first phase corresponds to a preset distance is... The first phase represents the true phase at the preset distance. The linear quantization function / piecewise linear function obtained after the aforementioned distance error correction can be used to determine the second phase corresponding to the distance. The second phase characterizes the phase calculation results of the time-of-flight image sensor. At this point, temperature drift correction can be performed using the following temperature drift correction function:

[0084]

[0085] Where a0 and a1 are the coefficients of temperature offset, which can be obtained by least squares fitting, and T is the operating temperature, which can be obtained from the temperature sensor of the time-of-flight image sensor.

[0086] In some embodiments of this application, the time-of-flight image sensor can also be calibrated by a light source. Specifically, the calibration image can include images captured at preset distances at multiple operating frequencies of the light source. In this case, such as... Figure 3 As shown, the parameter calibration method may further include steps S301 to S302.

[0087] Step S301: Determine the phase difference error caused by modulation frequency fluctuations based on the images corresponding to multiple operating frequencies.

[0088] Step S302: Determine the frequency correction function based on the phase difference error.

[0089] Among them, the frequency correction function can be used as a sensor parameter to correct the distance value.

[0090] The aforementioned multiple operating frequencies are different modulation frequencies of the light source, which can be set according to the actual situation. For example, they can be obtained by dividing the normal modulation frequency band of the light source into specified steps.

[0091] Specifically, considering the distance measurement error caused by the modulation frequency of the modulated light source, and interference from factors such as power supply ripple, the modulation frequency of the infrared modulated light source will fluctuate. Furthermore, due to circuit quantization, the frequency can only be selected as a specific discrete value, affecting the depth measurement value. Similar to temperature calibration, calibration images can be taken near the discrete values ​​corresponding to the normal operating frequency to fit the relationship between the operating frequency and the phase. Assuming the operating frequency (the light source's adjustment frequency) is f, and the nominal modulation frequency is f0 (i.e., the theoretical modulation frequency), the measured phase difference... Phase difference from actual The relationship between them is:

[0092]

[0093] in, The phase difference error caused by modulation frequency fluctuations can be fitted with the following formula:

[0094]

[0095] The coefficient k can be obtained by measuring the phase difference experimentally at different operating frequencies.

[0096] When finally performing depth calculations, distance correction can be performed according to the following frequency correction function:

[0097]

[0098] Where d is the distance. This refers to the phase difference, which can be the phase difference calculated using the aforementioned temperature correction. Δf requires the driver of the modulated light source to provide the difference between the current light source frequency and the set frequency. The set frequency is the modulation frequency of the light source specified by the user.

[0099] The calculated distance d can be applied to the aforementioned linear quantization function or piecewise linear function to obtain pixel values ​​and generate a depth image.

[0100] Figure 4 The illustration shows a schematic diagram of the implementation process of a depth image generation method provided in an embodiment of this application. This method can be applied to time-of-flight image sensors.

[0101] Specifically, the method for generating the aforementioned depth image may include the following steps S401 to S403.

[0102] Step S401: Obtain the sensor parameters obtained from the calibration of the time-of-flight image sensor.

[0103] Among them, the sensor parameters can be based on Figure 1-3 The parameter calibration method for the described time-of-flight image sensor may include at least a fitted curve or a piecewise linear function, and may also include one or more of the following: a temperature drift correction function and a frequency correction function.

[0104] Step S402: Determine the pixel value information of each pixel based on the sensor parameters.

[0105] In the embodiments of this application, the pixel value information of each pixel can be determined based on a fitted curve or a piecewise linear function. The pixel value information can characterize the depth of the corresponding pixel.

[0106] Specifically, the distance value of each pixel in the depth image can be determined, and the pixel value information can be determined based on the fitted curve or piecewise linear function in the sensor parameters.

[0107] In some embodiments of this application, the sensor parameters include a temperature drift correction function. In this case, determining the distance value of each pixel in the depth image may include: determining the corrected phase value based on the temperature drift correction function; and determining the distance value of each pixel based on the corrected phase value.

[0108] In some embodiments of this application, the sensor parameters include a frequency correction function; in this case, after determining the distance value of each pixel in the depth image, the method further includes: correcting the distance value according to the frequency correction function.

[0109] Specifically, a temperature drift correction function can be applied first to correct the phase values. Then, based on the corrected phase values, the distance values ​​for each pixel are determined. These distance values ​​are then corrected using a frequency correction function to obtain the corrected distance values. Substituting these corrected distance values ​​into a fitted curve or piecewise linear function allows the determination of the pixel value information for each pixel. This method corrects for the influence of temperature and light source on depth information, improving the reliability of the final depth image.

[0110] Step S403: Generate a first depth image based on pixel value information.

[0111] In the embodiments of this application, after obtaining the pixel value information of each pixel, a first depth image can be generated. At this time, the first depth image is a depth image obtained after distance error correction.

[0112] To further improve the reliability of depth images, such as Figure 5 As shown, after generating the first depth image based on the pixel value information, the method may further include steps S501 to S505.

[0113] Step S501: Acquire the intensity image captured by the time-of-flight image sensor.

[0114] The intensity image records the light intensity information of the reflected light corresponding to each pixel.

[0115] Step S502: Divide the first depth image and intensity image into n image block groups.

[0116] Each image block group consists of a depth image block and an intensity image block at the same pixel position, where n is greater than 1.

[0117] Specifically, the first depth image and the intensity image are the same size, and the positions of each image block are also one-to-one. For example, dividing the first depth image and the intensity image into a k*k format yields n = k*k depth image blocks and n = k*k intensity image blocks, respectively. Depth image blocks and intensity image blocks with the same pixel position form an image block group. For example, if the first depth image and the intensity image are divided into a 5*5 format, then the depth image block in the first row and first column and the intensity image block in the first row and first column form one image block group, the depth image block in the first row and second column and the intensity image block in the first row and second column form another image block group, and so on.

[0118] Step S503: Obtain the filtering parameters corresponding to each image block group.

[0119] The filtering parameters can be used to filter the pixel value information of each pixel in the first depth image.

[0120] In some embodiments of this application, the filtering parameters may include filtering coefficients and filtering offset. Obtaining the filtering parameters corresponding to each image patch group may include: for each image patch group, calculating the mean depth of the depth image patch and the mean intensity and variance of the intensity image patch; determining the filtering coefficients based on the intensity variance; and determining the filtering offset based on the filtering coefficients, the mean intensity, the mean depth, and a regularization parameter.

[0121] The mean is calculated for each depth image patch and each intensity image patch, using the following formulas:

[0122]

[0123] Where ω k Represents the set of pixels within block k, |ω k | represents the number of pixels within the block, I i The depth value of pixel i, G i μ represents the intensity value of pixel i. I,k μ represents the mean depth. G,k This represents the average intensity.

[0124] For intensity image patches, the intensity variance also needs to be calculated. The calculation formula is as follows:

[0125]

[0126] For each group of image patches, the linear filter coefficients a k and offset b k The formulas are as follows:

[0127]

[0128] b k =μ I,k -a k ·μ G,k ;

[0129] Here, ∈ is the regularization parameter, which can be specified by the user and is usually a very small value.

[0130] Step S504: For each image block group, determine the pixel value of each pixel based on the filtering parameters and the depth image blocks within the image block group.

[0131] For each image patch group, the user can apply the linear filtering coefficients a based on that image patch group. k and offset b k and the preset independent coefficient α k With β k The filtering strength is controlled by the pixel value Q of each pixel i within the image block group after filtering. i The calculation formula is:

[0132] Q i =α k ·(a k ·I i +b k )+β k .

[0133] Where, α k β can be set within the interval [0,2]. k The range can be consistent with the bit width of the output image.

[0134] Step S505: Generate a second depth image based on the pixel value of each pixel in the image block group.

[0135] In other words, by combining the depth information reflected by the pixel values ​​of each pixel within an image patch group, a second depth image can be regenerated. Compared to the first depth image, the second depth image takes into account the intensity image and filtering parameters. By enhancing the corrected first depth image based on the filtering parameters, the depth image can be made more refined, which helps to improve the reliability of the depth image.

[0136] In some embodiments of this application, generating a second depth image based on the pixel value of each pixel within an image block group may include: obtaining the weight of each pixel within each image block group; and generating a second depth image based on the weight and the pixel value of each pixel within the image block group.

[0137] Specifically, based on the weight ω k and the pixel value Q of each pixel within the image patch group k Fusion can be performed, and the fusion formula is: Q = ∑ k ω k ·Q k Wherein, ω k It is the weight of each pixel in the k-th image block group.

[0138] In some embodiments of this application, when dividing image blocks, there are overlapping regions between adjacent depth image blocks (shown in shaded areas), for example, please refer to... Figure 6 There is an overlapping region 61 between depth image block A and depth image block B.

[0139] At this point, the process of generating the second depth image based on weights and the pixel values ​​of each pixel within the image patch group may include: for a target pixel located in an overlapping region, determining the pixel value corresponding to each adjacent depth image patch based on weights and the pixel values ​​of each pixel within the image patch group; and performing a weighted average of the pixel values ​​corresponding to each adjacent depth image patch to obtain the pixel value of the target pixel in the second depth image.

[0140] In other words, a weighted average method can be used to handle overlapping parts between blocks. Please refer to [reference needed]. Figure 6 There is an overlapping region 61 between depth image block A and depth image block B. For pixels within the overlapping region 61, firstly, based on the weights and the pixel values ​​of each pixel within the image block group, determine the pixel value QA corresponding to depth image block A and the pixel value QB corresponding to depth image block B. Then, perform a weighted average of pixel values ​​QA and QB to obtain the pixel value of the target pixel in the second depth image. The above explanation uses two adjacent depth image blocks. In actual cases, some pixels may be located between four adjacent depth image blocks, but the calculation method is similar, that is, determine the pixel value corresponding to each depth image block and perform a weighted average. This application will not elaborate on this aspect.

[0141] When using a weighted average, the weights can be selected based on the actual situation. For example, they can be set to 0.5 respectively, or they can be set according to the distance between the pixel and the center point of each depth image block in the adjacent depth image block. The closer the distance, the higher the weight.

[0142] By setting overlapping portions during segmentation and performing weighted averaging during image fusion, boundary phenomena can be avoided during image block processing, which helps to make depth images smoother and more accurate.

[0143] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders.

[0144] like Figure 7 The diagram shown is a structural schematic of a parameter calibration device 700 for a time-of-flight image sensor provided in an embodiment of this application. The parameter calibration device 700 for the time-of-flight image sensor is configured on a calibration device.

[0145] Specifically, the parameter calibration device 700 for the time-of-flight image sensor may include:

[0146] The point determination unit 701 is used to determine the calibration point of the calibration image to be captured based on the linear interval satisfied by the intensity of the reflected light. The linear interval is obtained by dividing the range of the linear quantization function corresponding to the arctangent function.

[0147] Image acquisition unit 702 is used to acquire calibration images of each phase captured at the calibration point;

[0148] The calibration unit 703 is used to fit the pixel value of the calibration image corresponding to the phase at different distances for each pixel in the time-of-flight image sensor to obtain a fitting curve; the fitting curve represents the correspondence between distance and pixel value, and the fitting curve is used as sensor parameters for generating depth images.

[0149] In some embodiments of this application, the calibration unit 703 may be specifically used to: segment the fitted curve according to distance intervals to obtain a piecewise linear function, wherein the linear function corresponding to different distance intervals in the piecewise linear function is different.

[0150] In some embodiments of this application, at each calibration point, there are multiple calibration images for each phase; the image acquisition unit 702 may also be specifically used to: remove abnormal images from the calibration images; for each phase at the calibration point, take the average value of the remaining calibration images after removing abnormal images to obtain an average image.

[0151] In some embodiments of this application, the calibration image includes images captured at a preset distance under multiple operating temperatures of the time-of-flight image sensor; the calibration unit 703 may be specifically used to: determine a first phase corresponding to the preset distance; determine a second phase based on the images corresponding to the multiple operating temperatures and the linear quantization function; and determine a temperature drift correction function based on the first phase, the second phase, and the multiple operating temperatures, wherein the temperature drift correction function is used as a sensor parameter to correct the phase value.

[0152] In some embodiments of this application, the calibration image includes images captured at a preset distance at multiple operating frequencies of the light source; the calibration unit 703 may be specifically used to: determine the phase difference error caused by modulation frequency fluctuation based on the images corresponding to the multiple operating frequencies respectively; and determine a frequency correction function based on the phase difference error, wherein the frequency correction function is used as a sensor parameter to correct the distance value.

[0153] It should be noted that, for the sake of convenience and brevity, the specific working process of the parameter calibration device 700 for the aforementioned time-of-flight image sensor can be found in [reference needed]. Figures 1 to 3 The corresponding process of the method will not be described in detail here.

[0154] like Figure 8 The diagram shown is a schematic diagram of a depth image generation device 800 provided in an embodiment of this application. The depth image generation device 800 is disposed on a time-of-flight image sensor.

[0155] Specifically, the depth image generation device 800 may include:

[0156] The parameter acquisition unit 801 is used to acquire sensor parameters obtained by calibrating the time-of-flight image sensor, wherein the sensor parameters are obtained according to the parameter calibration method of the time-of-flight image sensor described in the first aspect.

[0157] The pixel value determination unit 802 is used to determine the pixel value information of each pixel based on the sensor parameters.

[0158] The image generation unit 803 is used to generate a first depth image based on the pixel value information.

[0159] In some embodiments of this application, the pixel value determination unit 802 may be specifically used to: determine the distance value of each pixel in the depth image; and determine the pixel value information based on the fitting curve or piecewise linear function in the sensor parameters.

[0160] In some embodiments of this application, the sensor parameters include a temperature drift correction function; the pixel value determination unit 802 may be specifically used to: determine the corrected phase value according to the temperature drift correction function; and determine the distance value of each pixel according to the corrected phase value.

[0161] In some embodiments of this application, the sensor parameters include a frequency correction function; the pixel value determination unit 802 may be specifically used to: correct the distance value according to the frequency correction function.

[0162] In some embodiments of this application, the image generation unit 803 may also be used to: acquire an intensity image captured by the time-of-flight image sensor; divide the first depth image and the intensity image into n image block groups, each image block group including a depth image block and an intensity image block at the same pixel position, where n is greater than 1; acquire filtering parameters corresponding to each image block group; for each image block group, determine the pixel value of each pixel based on the filtering parameters and the depth image block within the image block group; and generate a second depth image based on the pixel value of each pixel within the image block group.

[0163] In some embodiments of this application, the filtering parameters include filtering coefficients and filtering offset; the image generation unit 803 can also be used to: for each of the image block groups, calculate the depth mean of the depth image block and the intensity mean and intensity variance of the intensity image block; determine the filtering coefficients based on the intensity variance; and determine the filtering offset based on the filtering coefficients, the intensity mean, the depth mean, and the regularization parameter.

[0164] In some embodiments of this application, the image generation unit 803 may also be used to: obtain the weight of each pixel in each of the image block groups; and generate the second depth image according to the weight and the pixel value of each pixel in the image block group.

[0165] In some embodiments of this application, there is an overlapping region between adjacent depth image blocks; the image generation unit 803 can also be used to: determine the pixel value corresponding to the adjacent depth image blocks for the target pixel located in the overlapping region according to the weight and the pixel value of each pixel in the image block group; and perform a weighted average of the pixel values ​​corresponding to the adjacent depth image blocks to obtain the pixel value of the target pixel in the second depth image.

[0166] It should be noted that, for the sake of convenience and brevity, the specific working process of the depth image generation device 800 described above can be found in [reference needed]. Figures 4 to 6 The corresponding process of the method will not be described in detail here.

[0167] like Figure 9 The diagram shown is a schematic of a calibration device provided in an embodiment of this application. Specifically, the calibration device 9 may include: a processor 90, a memory 91, and a computer program 92 stored in the memory 91 and executable on the processor 90, such as a parameter calibration program for a time-of-flight image sensor.

[0168] When the processor 90 executes the computer program 92, it implements the steps in the above-described embodiments of the parameter calibration methods for various time-of-flight image sensors, for example... Figure 1 Steps S101 to S103 are shown. Alternatively, when the processor 90 executes the computer program 92, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 7 The functions of the point determination unit 701, image acquisition unit 702, and calibration unit 703 are shown.

[0169] The computer program can be divided into one or more modules / units, which are stored in the memory 91 and executed by the processor 90 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the calibration device.

[0170] The calibration device may include, but is not limited to, a processor 90 and a memory 91. Those skilled in the art will understand that... Figure 9 This is merely an example of a calibration device and does not constitute a limitation on the calibration device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the calibration device may also include input / output devices, network access devices, buses, etc.

[0171] The processor 90 referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0172] The memory 91 can be an internal storage unit of the calibration device, such as the hard drive or memory of the calibration device. The memory 91 can also be an external storage device of the calibration device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the calibration device. Furthermore, the memory 91 can include both internal and external storage units of the calibration device. The memory 91 is used to store the computer program and other programs and data required by the calibration device. The memory 91 can also be used to temporarily store data that has been output or will be output.

[0173] It should be noted that, for the sake of convenience and brevity, the structure of the above calibration device can also be referred to the specific description of the structure in the method embodiment, which will not be repeated here.

[0174] like Figure 10The diagram shown is a schematic of a time-of-flight image sensor provided in an embodiment of this application. Specifically, the time-of-flight image sensor 10 may include: a processor 100, a memory 101, and a computer program 102 stored in the memory 101 and executable on the processor 100, such as a depth image generation program.

[0175] When the processor 100 executes the computer program 102, it implements the steps in the above-described embodiments of the methods for generating depth images, for example... Figure 4 Steps S401 to S403 are shown. Alternatively, when the processor 100 executes the computer program 102, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 8 The functions of the parameter acquisition unit 801, pixel value determination unit 802, and image generation unit 803 are shown.

[0176] The computer program can be divided into one or more modules / units, which are stored in the memory 101 and executed by the processor 100 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the calibration device.

[0177] It should be noted that, for the sake of convenience and brevity, the structure of the above-mentioned time-of-flight image sensor can also be referred to the specific description of the structure in the method embodiment, which will not be repeated here.

[0178] like Figure 11 As shown, this application also provides a time-of-flight image sensor system, including: a light source, a time-of-flight image sensor, a light source driver chip, and a lens. The time-of-flight image sensor is used to perform actions such as... Figure 4-6 The steps of the method for generating the depth image.

[0179] The aforementioned light source can be an infrared light source. The product forms of the aforementioned time-of-flight image sensor system include, but are not limited to, modules, printed circuit boards (PCBs), etc.

[0180] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0181] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0182] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for various specific applications, but such implementations should not be considered beyond the scope of this application.

[0183] In the embodiments provided in this application, it should be understood that the disclosed apparatus / calibration devices and methods can be implemented in other ways. For example, the apparatus / calibration device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0184] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0186] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0187] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for calibrating parameters of a time-of-flight image sensor, characterized in that, include: The calibration points for capturing calibration images are determined based on the linear interval satisfied by the intensity of reflected light. The linear interval is obtained by dividing the range of the linear quantization function corresponding to the arctangent function. Acquire calibration images of each phase taken at the calibration point; For each pixel in the time-of-flight image sensor, a fitting curve is obtained by fitting the pixel value of the corresponding phase calibration image at different distances; the fitting curve represents the correspondence between distance and pixel value, and the fitting curve is used as the sensor parameter for generating depth images.

2. The parameter calibration method for a time-of-flight image sensor as described in claim 1, characterized in that, After obtaining the fitted curve, the parameter calibration method further includes: The fitted curve is segmented according to distance intervals to obtain a piecewise linear function, in which different distance intervals correspond to different linear functions.

3. The parameter calibration method for a time-of-flight image sensor as described in claim 1, characterized in that, At each calibration point, there are multiple calibration images for each phase; After acquiring calibration images of each phase taken at the calibration point, the parameter calibration method further includes: Remove abnormal images from the calibration images; For each phase at the calibration point, the average value of the remaining calibration images after removing abnormal images is taken to obtain the average image.

4. The parameter calibration method for a time-of-flight image sensor as described in any one of claims 1-3, characterized in that, The calibration images include images captured at a preset distance under multiple operating temperatures of the time-of-flight image sensor; The parameter calibration method further includes: Determine the first phase corresponding to the preset distance; The second phase is determined based on the images corresponding to the multiple operating temperatures and the linear quantization function. Based on the first phase, the second phase, and the plurality of operating temperatures, a temperature drift correction function is determined, which is used as a sensor parameter to correct the phase value.

5. The parameter calibration method for a time-of-flight image sensor as described in any one of claims 1-3, characterized in that, The calibration images include images captured at preset distances at multiple operating frequencies of the light source; The parameter calibration method further includes: Based on the images corresponding to the multiple operating frequencies, determine the phase difference error caused by modulation frequency fluctuations; Based on the phase difference error, a frequency correction function is determined, which is used as a sensor parameter to correct the distance value.

6. A method for generating a depth image, characterized in that, include: Obtain sensor parameters obtained from the calibration of the time-of-flight image sensor, wherein the sensor parameters are obtained by the parameter calibration method of the time-of-flight image sensor according to any one of claims 1-5; Based on the sensor parameters, determine the pixel value information for each pixel. A first depth image is generated based on the pixel value information.

7. The method for generating a depth image as described in claim 6, characterized in that, The step of determining the pixel value information of each pixel based on the sensor parameters includes: Determine the distance value of each pixel in the depth image; The pixel value information is determined based on the fitted curve or piecewise linear function in the sensor parameters.

8. The method for generating a depth image as described in claim 7, characterized in that, The sensor parameters include a temperature drift correction function; Determining the distance value of each pixel in the depth image includes: The corrected phase value is determined based on the temperature drift correction function; Based on the corrected phase value, the distance value of each pixel is determined.

9. The method for generating a depth image as described in claim 7, characterized in that, The sensor parameters include a frequency correction function; After determining the distance values ​​of each pixel in the depth image, the generation method further includes: The distance value is corrected according to the frequency correction function.

10. The method for generating a depth image as described in any one of claims 6-9, characterized in that, After generating the first depth image based on the pixel value information, the generation method further includes: Acquire the intensity image captured by the time-of-flight image sensor; The first depth image and the intensity image are divided into n image block groups. Each image block group includes a depth image block and an intensity image block with the same pixel position, where n is greater than 1. Obtain the filtering parameters corresponding to each of the image block groups; For each image patch group, the pixel value of each pixel is determined based on the filtering parameters and the depth image patch within the image patch group; A second depth image is generated based on the pixel value of each pixel within the image block group.

11. The method for generating a depth image as described in claim 10, characterized in that, The filtering parameters include filtering coefficients and filtering offset; The step of obtaining the filtering parameters corresponding to each image block group includes: For each group of image patches, calculate the mean depth of the depth image patch and the mean intensity and variance of the intensity image patch; The filtering coefficients are determined based on the intensity variance. The filter offset is determined based on the filter coefficients, the mean intensity, the mean depth, and the regularization parameter.

12. The method for generating a depth image as described in claim 10, characterized in that, The step of generating a second depth image based on the pixel value of each pixel within the image block group includes: Obtain the weight of each pixel within each of the image block groups; The second depth image is generated based on the weights and the pixel values ​​of each pixel within the image block group.

13. The method for generating a depth image as described in claim 12, characterized in that, There are overlapping regions between adjacent depth image blocks; The process of generating the second depth image based on the weights and the pixel values ​​of each pixel within the image patch group includes: For a target pixel located in the overlapping region, the pixel value corresponding to the adjacent depth image block is determined according to the weight and the pixel value of each pixel in the image block group; The pixel value of the target pixel in the second depth image is obtained by weighted averaging the pixel values ​​corresponding to the adjacent depth image blocks.

14. A calibration device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the parameter calibration method for the time-of-flight image sensor as described in any one of claims 1 to 5.

15. A time-of-flight image sensor, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method for generating a depth image as described in any one of claims 6 to 13.

16. A time-of-flight image sensor system, characterized in that, include: The system includes a light source, a time-of-flight image sensor, a light source driver chip, and a lens, wherein the time-of-flight image sensor is used to perform the steps of the depth image generation method as described in any one of claims 6 to 13.

17. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the parameter calibration method for the time-of-flight image sensor as described in any one of claims 1 to 5, or, when the computer program is executed by the processor, it implements the steps of the depth image generation method as described in any one of claims 6 to 13.

Images