Depth image processing method and apparatus

Abstract

The embodiment of the present application provides a kind of depth image processing method and equipment, the method includes obtaining the first depth image of first object;The depth value of pixel in the first depth image is corrected by field phase correction coefficient matrix to obtain second depth image, wherein, the field phase correction coefficient matrix is the matrix for correcting field phase error obtained after the preprocessing of n initial depth images, the n initial depth images are the depth image obtained in first condition, the first condition is the start time of the light source of shooting device and the start time of the sensor of the shooting device are different time, the preprocessing includes the mean processing made according to the n initial depth images, and n is the integer greater than 1.It can reduce the distortion rate of depth image by using the embodiment of the present application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a depth image processing method and device. BACKGROUND

[0002] A depth image refers to an image taking vertical distance (depth) from an image collector to each point in a scene as a pixel value, which directly reflects the geometric shape of a visible surface of a scene. Methods for obtaining a depth image include a laser radar depth imaging method, a computer stereo vision imaging method, a coordinate measuring machine method, a Moire fringe method, a structured light method, etc. A depth image is a three-dimensional representation of an object, which is generally obtained through a stereo camera or a time of flight (TOF) camera, etc. However, a depth image obtained through a TOF camera has errors, and the main sources of the errors are as follows:

[0003] Depth nonlinear error: when an active light source and a TOF sensor emit waves that are not perfect sinusoidal waves, the depth value measured by the TOF will have nonlinear error. The nonlinear error has different error values at different distances. Taking a square wave as an example: the error amount is similar to that of a sinusoidal wave, which oscillates with distance and has periodic repetition.

[0004] Field of view phase error: a time ranging system uses time of flight difference to calculate distance, so the measured information is actually the straight-line distance between an image collector and a pixel point, rather than the depth of a Z-axis, that is, the vertical distance between the image collector and the pixel point. The field of view phase error is the error between the vertical distance and the straight-line distance.

[0005] In the prior art, there is a technical solution for calculating a field of view using lens parameters, and correcting the field of view phase error using a cosine value. However, in this solution, only the field of view phase error is corrected, and the residual depth nonlinear error is not considered, so the distortion rate of the obtained depth image is high. In summary, how to correct the field of view phase error of a depth image while reducing the depth nonlinear error to reduce the distortion rate of the depth image is a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0006] The present application provides a depth image processing method and device, which can correct the field of view phase error of a depth image while reducing the depth nonlinear error to reduce the distortion rate of the depth image.

[0007] In a first aspect, the present application provides a depth image processing method, which comprises:

[0008] obtaining a first depth image of a first object; and correcting a depth value of a pixel in the first depth image by a field phase correction coefficient matrix to obtain a second depth image, wherein the field phase correction coefficient matrix is a matrix for correcting a field phase error obtained by preprocessing n initial depth images, the n initial depth images are depth images obtained in a first condition, the first condition includes that a starting time of a light source of a photographing device and a starting time of a sensor of the photographing device are different time points, the preprocessing includes mean processing performed according to the n initial depth images, and n is an integer greater than 1.

[0009] Specifically, the field phase correction coefficient matrix is used to reduce the field phase error of the depth value of each pixel in the first depth image. The field phase error is an error caused by an angle deviation of a field of view.

[0010] The n initial depth images have the same size, that is, the pixel matrixes of the n initial depth images are matrices with equal number of rows and equal number of columns.

[0011] The execution subject of the depth image processing method provided in the present application can be the photographing device or another photographing device other than the photographing device, and the photographing device or the other photographing device stores data of the field phase correction coefficient matrix.

[0012] It should be noted that, in the present application, the starting time of the light source can also be referred to as the time when the light source starts to emit a light signal, and the starting time of the sensor can also be referred to as the time when the sensor starts to be able to receive a reflected light signal.

[0013] The depth nonlinear error is a nonlinear error caused by the fact that the waveform of the light signal is not a standard waveform (for example, is not a standard sine wave, etc.), and the error amount of the nonlinear error oscillates with distance changes, with positive and negative error amounts. Therefore, in the present application, by sampling multiple times in the case where the light source of the photographing device is started later than the sensor, and by mean processing performed according to the n initial depth images, the positive and negative error amounts in the nonlinear error can offset each other, so that the depth nonlinear error introduced in the measurement process can be reduced. Meanwhile, the field phase correction coefficient matrix obtained based on the sampling and the mean processing can correct the depth value of each pixel in the depth image, reduce the error caused by the field of view phase, and greatly reduce the distortion rate of the finally obtained depth image, thereby improving the quality of the depth image.

[0014] In one possible implementation, the preprocessing further includes surface fitting processing, and the field phase correction coefficient matrix is calculated based on n fitted depth images, which are obtained by surface fitting on the n initial depth images of the second object obtained by the n times of TOF ranging based on the shooting device, the starting time of the light source in the i-th time is delayed by (i-1)*Δt from the starting time of the sensor, i is in the range of [1, n], n*|Δt|=k*T, T is the period of the light signal, k is an integer, and Δt is a preset time length.

[0015] n*|Δt|=k*T indicates that the total amount of time of n times of delay is an integer multiple of the period of the light signal emitted by the light source. The sizes of the n fitted depth images are the same, that is, the pixel matrices of the n fitted depth images are matrices with equal number of rows and equal number of columns. In addition, the sizes of the n fitted depth images are the same as the sizes of the n initial depth images, that is, the pixel matrices of the n initial depth images and the pixel matrices of the n fitted depth images are matrices with equal number of rows and equal number of columns.

[0016] Optionally, the surface fitting method can be polynomial surface fitting of least squares method or surface fitting method of other adaptive function.

[0017] It should be noted that the starting time of the light source delayed from the starting time of the sensor can also be referred to as the time when the light source starts to emit the light signal delayed from the time when the sensor starts to receive the reflected light signal.

[0018] In this application, the two characteristics of periodic oscillation and zero sum (the total error in a complete cycle is zero) of depth nonlinear error are used, the depth images are sampled multiple times under the condition that the light source of the shooting device is delayed from the sensor, the total time of multiple delays is an integer multiple of the period of the sampling light signal, and then the pixel matrices of the depth images obtained by multiple delays are averaged to reduce the depth nonlinear error, thereby improving the accuracy of the obtained field phase correction coefficient matrix.

[0019] In one possible implementation, the initial depth image obtained in the i-th time is the i-th initial depth image, which is represented by a pixel matrix Di(x, y), and the pixel matrix Di(x, y) is the sum of each pixel value in the pixel matrix Di(x, y)' and c*[(1-i)*Δt] / 2, Di(x, y)' is the pixel matrix of the depth image obtained in the i-th time, and c*[(1-i)*Δt] / 2 is the distance difference caused by the delay of the starting time of the light source from the starting time of the sensor by (i-1)*Δt.

[0020] In the present application, the operation of increasing the depth by c*[(1-i)*Δt] / 2 after the delay time (i-1)*Δt is used is to compensate the distance difference caused by the delay time with the depth difference of the theoretical value of the time ranging, so that the depth non-linear error amount from different time phase points of the same pixel is obtained at the same time as the initial depth image.

[0021] In one possible implementation, the field of view phase correction coefficient matrix is calculated according to the n fitting depth images, and includes: the field of view phase correction coefficient matrix is a ratio matrix obtained by taking the ratio of the minimum value of each pixel value in the fitting average pixel matrix to the respective pixel value, and the fitting average pixel matrix is obtained by averaging the pixel matrix of the n fitting depth images.

[0022] In the present application, the field of view phase correction coefficient matrix is calculated, so that the field of view phase error of the depth image obtained by the TOF ranging method of the shooting device can be corrected.

[0023] In one possible implementation, the depth image processing method further includes: correcting the second depth image by a fixed pattern noise (FPN) matrix to obtain a third depth image, wherein each value in the FPN matrix includes a fixed noise of a corresponding pixel in the first depth image caused by hardware. It should be noted that the present application can be implemented in combination with the method in the first aspect and one or more of the possible implementations.

[0024] The execution subject of the depth image processing method provided in the present application can be the shooting device or other shooting devices other than the shooting device, and the data of the FPN matrix is stored in the shooting device or the other shooting device. Specifically, the fixed pattern noise FPN can also be called pixel fixed noise. Due to the differences in manufacturing process, reading order and circuit design, different pixels will introduce errors between pixels. In the present application, in addition to being able to correct the field of view phase error of the depth image, the fixed pattern noise of each pixel caused by the hardware of the shooting device can also be corrected. In one possible implementation, the FPN matrix is a difference matrix obtained by taking the difference between a fitting average pixel matrix and an initial average pixel matrix, the fitting average pixel matrix is obtained by averaging the pixel matrix of the n fitting depth images, and the initial average pixel matrix is obtained by averaging the pixel matrix of the n initial depth images.

[0025] In the present application, the fixed pattern noise matrix is obtained through the calculation process, so that it can be used to reduce the fixed pattern noise of the depth image.

[0026] In one possible implementation, the second depth image is obtained by correcting the first depth image using the field phase correction coefficient matrix, including: calculating the product of the pixel matrix of the first depth image and the pixel value of the same subscript in the field phase correction coefficient matrix to obtain the second depth image.

[0027] The size of the first depth image can be the same as the size of the field phase correction coefficient matrix, for example, assuming that the size of the field phase correction coefficient matrix S1(x, y) is 1024*1024, the size of the pixel matrix of the first depth image is also 1024*1024.

[0028] The application provides a calculation process for correcting field phase error.

[0029] In one possible implementation, the third depth image is obtained by correcting the second depth image using the fixed pattern noise (FPN) matrix, including: calculating the product of the pixel matrix of the first depth image and the pixel value of the same subscript in the field phase correction coefficient matrix to obtain a product matrix; and calculating the sum of the product matrix and the FPN matrix to obtain the third depth image.

[0030] The size of the first depth image, the size of the FPN matrix, and the size of the field phase correction coefficient matrix are the same, for example, assuming that the size of the field phase correction coefficient matrix is 1024*1024, the size of the pixel matrix of the first depth image and the size of the FPN matrix are also 1024*1024.

[0031] The application provides a calculation process for simultaneously correcting fixed pattern noise and field phase error.

[0032] In a second aspect, the application provides a depth image processing method, including: obtaining n initial depth images, the n initial depth images being depth images of a second object obtained by a shooting device through time-of-flight (TOF) ranging method n times, wherein the starting time of a light source of the shooting device in the i-th time is delayed from the starting time of a sensor of the shooting device by (i-1)*Δt, the value range of i is [1, n], n*|Δt|=k*T, T is the period of the light signal, n is an integer greater than 1, k is an integer, and Δt is a preset time length;

[0033] Curved surface fitting is performed on the n initial depth images to obtain n fitted depth images;

[0034] The field phase correction coefficient matrix is used to correct the field phase error of the depth image obtained by the shooting device through the TOF ranging method.

[0035] The second object can be a flat plane.

[0036] In the present application, the periodic oscillation and zero-sum (the total error in a complete cycle is zero) characteristics of the depth nonlinear error are utilized. The depth image is sampled multiple times with the light source of the shooting device being delayed from the sensor, and the total time of the multiple delays is an integer multiple of the period of the sampled light signal. Then, the pixel matrix of the depth image obtained through the multiple delays is averaged to reduce the depth nonlinear error, thereby improving the accuracy of the obtained field phase correction coefficient matrix. Therefore, the field phase correction coefficient matrix of the depth image obtained by the present application can correct the field phase error of each pixel value in the depth image obtained by the shooting device, thereby reducing the distortion rate of the depth image and improving the quality of the depth image.

[0037] In one possible implementation, the initial depth image obtained in the i-th time is the i-th initial depth image; and the n initial depth images are obtained by:

[0038] The n measurement depth images actually measured by the shooting device through the TOF ranging method are obtained, and the i-th measurement depth image in the n measurement depth images is represented by a pixel matrix Di(x, y)';

[0039] The sum of each pixel value in the pixel matrix Di(x, y)' and c*[(1-i)*Δt] / 2 is calculated to obtain the i-th initial depth image Di(x, y), and c*[(1-i)*Δt] / 2 is the distance difference caused by the delay of (i-1)*Δt between the starting time of the light source and the starting time of the sensor.

[0040] In the present application, the operation of increasing the depth by c*[(1-i)*Δt] / 2 after the delay time (i-1)*Δt is used to compensate the distance difference caused by the delay time with the depth difference of the time ranging theory value, so that the depth nonlinear error amount of the same pixel from different time phase points is obtained at the same time.

[0041] In one possible implementation, the field phase correction coefficient matrix is calculated after the average processing of the n fitting depth images, and the calculation includes:

[0042] The pixel matrices of the n fitting depth images are averaged to obtain a fitting average pixel matrix;

[0043] extracting a minimum value of a plurality of pixel values in the fitted average pixel matrix, and calculating a ratio matrix by calculating a ratio of the minimum value to each pixel value in the fitted average pixel matrix, the ratio matrix being the field of view phase correction coefficient matrix.

[0044] In the present application, the field of view phase correction coefficient matrix is calculated, so that the field of view phase error of the depth image obtained by the TOF ranging method can be corrected.

[0045] In one possible implementation, after the surface fitting of the n initial depth images to obtain n fitted depth images, the method further comprises:

[0046] averaging the pixel matrices of the n initial depth images to obtain an initial average pixel matrix;

[0047] averaging the pixel matrices of the n fitted depth images to obtain a fitted average pixel matrix;

[0048] calculating a difference matrix by calculating a difference between the fitted average pixel matrix and the initial average pixel matrix, the difference matrix being a fixed pattern noise (FPN) matrix, each value in the FPN matrix including a fixed noise of a pixel in the depth image with the same subscript as the value caused by hardware.

[0049] In the present application, the fixed pattern noise matrix is obtained through the calculation process, so that it can be used to reduce the fixed pattern noise of the depth image.

[0050] In a third aspect, the present application provides a depth image processing device, which comprises:

[0051] an acquisition unit configured to acquire a first depth image of a first object;

[0052] a correction unit configured to correct a depth value of a pixel in the first depth image by a field of view phase correction coefficient matrix to obtain a second depth image, wherein the field of view phase correction coefficient matrix is a matrix for correcting a field of view phase error obtained after preprocessing n initial depth images, the n initial depth images being depth images obtained in a first condition, the first condition including that a starting time of a light source of a shooting device and a starting time of a sensor of the shooting device are different time points, the preprocessing including mean value processing performed according to the n initial depth images, the n being an integer greater than 1.

[0053] In one possible implementation, the preprocessing further comprises surface fitting processing,

[0054] The field-of-view phase correction coefficient matrix is calculated according to n fitting depth images, the n fitting depth images are obtained by respectively performing surface fitting on the n initial depth images, the n initial depth images are depth images of the second object obtained by the n times of TOF ranging based on the shooting device, the starting time of the light source in the i-th time is delayed by (i-1)*Δt from the starting time of the sensor, the value range of i is [1, n], n*|Δt|=k*T, T is the period of the light signal, k is an integer, and Δt is a preset time length.

[0055] In one possible implementation, the initial depth image obtained in the i-th time is the i-th initial depth image, the i-th initial depth image is represented by a pixel matrix Di(x, y), each pixel value in the pixel matrix Di(x, y) is obtained by taking the sum of c*[(1-i)*Δt] / 2, Di(x, y)' is a pixel matrix of the depth image obtained by the i-th measurement, and c*[(1-i)*Δt] / 2 is a distance difference caused by the delay of the starting time of the light source from the starting time of the sensor by (i-1)*Δt.

[0056] In one possible implementation, the field-of-view phase correction coefficient matrix is calculated according to n fitting depth images, including:

[0057] The field-of-view phase correction coefficient matrix is a ratio matrix obtained by respectively taking the ratio of the minimum value of a plurality of pixel values in a fitting average pixel matrix and each pixel value of the fitting average pixel matrix, and the fitting average pixel matrix is obtained by averaging the pixel matrices of the n fitting depth images.

[0058] In one possible implementation, the correction unit is further configured to:

[0059] A third depth image is obtained by correcting the second depth image through a fixed pattern noise (FPN) matrix, wherein each value in the FPN matrix includes a fixed noise of a pixel in the first depth image with the same subscript as the value caused by hardware.

[0060] In one possible implementation, the FPN matrix is a difference matrix obtained by taking the difference between a fitting average pixel matrix and an initial average pixel matrix, the fitting average pixel matrix is obtained by averaging the pixel matrices of the n fitting depth images, and the initial average pixel matrix is obtained by averaging the pixel matrices of the n initial depth images.

[0061] In one possible implementation, the correction unit is specifically configured to:

[0062] The product matrix is obtained by respectively calculating the product of the pixel matrix of the first depth image and the pixel value with the same subscript in the field phase correction coefficient matrix.

[0063] In one possible implementation, the correction unit is specifically configured to:

[0064] The product matrix is obtained by respectively calculating the product of the pixel matrix of the first depth image and the pixel value with the same subscript in the field phase correction coefficient matrix.

[0065] The third depth image is obtained by calculating the sum of the product matrix and the FPN matrix.

[0066] In a fourth aspect, the present application provides a depth image processing device, which comprises:

[0067] The acquisition unit is configured to acquire n initial depth images, the n initial depth images being depth images of a second object acquired by a shooting device through a time-of-flight (TOF) ranging method n times, wherein the starting time of a light source of the shooting device in the i-th time of the n times is delayed from the starting time of a sensor of the shooting device by (i-1)*Δt, i is in the range of [1, n], n*|Δt|=k*T, T is the period of the light signal, n is an integer greater than 1, k is an integer, and Δt is a preset time length.

[0068] The curved surface fitting unit is configured to perform curved surface fitting on the n initial depth images respectively to obtain n fitted depth images; and the calculation unit is configured to calculate a field phase correction coefficient matrix according to the n fitted depth images after mean value processing, the field phase correction coefficient matrix being used to correct the field phase error of a depth image acquired by the shooting device through the TOF ranging method.

[0069] In one possible implementation, the initial depth image obtained in the i-th time is the i-th initial depth image; and the acquisition unit is specifically configured to:

[0070] The acquisition unit is configured to acquire n initial depth images, the n initial depth images being depth images of a second object acquired by a shooting device through a time-of-flight (TOF) ranging method n times, wherein the starting time of a light source of the shooting device in the i-th time of the n times is delayed from the starting time of a sensor of the shooting device by (i-1)*Δt, i is in the range of [1, n], n*|Δt|=k*T, T is the period of the light signal, n is an integer greater than 1, k is an integer, and Δt is a preset time length.

[0071] The i-th initial depth image Di(x, y) is obtained by respectively calculating the sum of each pixel value in the pixel matrix Di(x, y)' and c*[(1-i)*Δt] / 2, c*[(1-i)*Δt] / 2 being the distance difference caused by the delay of the starting time of the light source from the starting time of the sensor by (i-1)*Δt.

[0072] In one possible implementation, the calculation unit is specifically configured to:

[0073] averaging the pixel matrices of the n fitted depth images to obtain a fitted average pixel matrix;

[0074] extracting a minimum value of the pixel values in the fitted average pixel matrix, and calculating a ratio matrix by calculating a ratio of the minimum value to each pixel value in the fitted average pixel matrix, the ratio matrix being the field phase correction coefficient matrix. In one possible implementation, the calculation unit is further configured to, after the surface fitting unit performs surface fitting on the n initial depth images to obtain n fitted depth images,

[0075] averaging the pixel matrices of the n initial depth images to obtain an initial average pixel matrix;

[0076] averaging the pixel matrices of the n fitted depth images to obtain a fitted average pixel matrix;

[0077] calculating a difference matrix by calculating a difference between the fitted average pixel matrix and the initial average pixel matrix, the difference matrix being a fixed pattern noise (FPN) matrix, each value in the FPN matrix including a fixed noise of a pixel in a depth image with a same subscript as the value caused by hardware.

[0078] In a fifth aspect, the present application provides a depth image processing device, which comprises a processor, a communication interface and a memory, wherein the memory, the communication interface and the processor can be integrated together or coupled through a coupler, the memory is configured to store a computer program, and the processor is configured to execute the computer program stored in the memory to implement the following operations:

[0079] obtaining a first depth image of a first object; and correcting a depth value of a pixel in the first depth image by using a field phase correction coefficient matrix to obtain a second depth image, wherein the field phase correction coefficient matrix is a matrix obtained by preprocessing n initial depth images, the n initial depth images being depth images obtained in a first case, the first case including a case that a starting time of a light source of a photographing device is different from a starting time of the photographing device, and the preprocessing including mean value processing performed on the n initial depth images, n being an integer greater than 1.

[0080] The execution subject of the depth image processing method can be the photographing device or another photographing device, and the photographing device or the other photographing device stores data of the field phase correction coefficient matrix.

[0081] In a possible implementation, the preprocessing further includes surface fitting processing, and the field-of-view phase correction coefficient matrix is calculated according to n fitted depth images, the n fitted depth images are respectively obtained by surface fitting on the n initial depth images, the n initial depth images are depth images of the second object obtained by the n times of TOF ranging based on the shooting device, and the starting time of the light source in the i th time is delayed by (i-1)*Δt from the starting time of the sensor, i is in the range of [1, n], n*|Δt|=k*T, T is the period of the light signal, k is an integer, and Δt is a preset time length.

[0082] Optionally, the surface fitting method can be polynomial surface fitting of least square method or surface fitting method of other adaptive function.

[0083] In a possible implementation, the i th obtained initial depth image is the i th initial depth image, the i th initial depth image is represented by a pixel matrix Di(x, y), each pixel value in the pixel matrix Di(x, y) is obtained by summing c*[(1-i)*Δt] / 2, Di(x, y)' is a pixel matrix of the depth image obtained by the i th measurement, and c*[(1-i)*Δt] / 2 is a distance difference caused by the delay of the starting time of the light source from the starting time of the sensor by (i-1)*Δt.

[0084] In a possible implementation, the field-of-view phase correction coefficient matrix is calculated according to n fitted depth images, and the field-of-view phase correction coefficient matrix is a ratio matrix obtained by respectively taking the minimum value of a plurality of pixel values in a fitted average pixel matrix as a ratio of each pixel value in the fitted average pixel matrix, and the fitted average pixel matrix is obtained by averaging pixel matrices of the n fitted depth images.

[0085] In a possible implementation, the depth image processing method further includes: obtaining a third depth image by correcting the second depth image through a fixed pattern noise (FPN) matrix, wherein each value in the FPN matrix respectively includes a fixed noise of a corresponding pixel in the first depth image caused by hardware. It should be noted that the present application can be implemented in combination with the method in the first aspect and / or the method in the possible implementation.

[0086] In a possible implementation, the FPN matrix is a difference matrix obtained by taking a difference between a fitted average pixel matrix and an initial average pixel matrix, the fitted average pixel matrix is obtained by averaging pixel matrices of the n fitted depth images, and the initial average pixel matrix is obtained by averaging pixel matrices of the n initial depth images.

[0087] In one possible implementation, the second depth image is obtained by correcting the first depth image using the field phase correction coefficient matrix, including: calculating the product of the pixel matrix of the first depth image and the pixel value of the same subscript in the field phase correction coefficient matrix to obtain the second depth image.

[0088] In one possible implementation, the third depth image is obtained by correcting the second depth image using the fixed pattern noise (FPN) matrix, including: calculating the product of the pixel matrix of the first depth image and the pixel value of the same subscript in the field phase correction coefficient matrix to obtain a product matrix; and calculating the sum of the product matrix and the FPN matrix to obtain the third depth image.

[0089] In a sixth aspect, the present application provides a depth image processing device, which comprises a processor, a communication interface and a memory, wherein the memory, the communication interface and the processor can be integrated together or coupled through a coupler, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to realize the following operations: obtaining n initial depth images, the n initial depth images being depth images of a second object obtained by a shooting device through a time of flight (TOF) ranging method n times, wherein the starting time of a light source of the shooting device in the i th time is delayed from the starting time of a sensor of the shooting device by (i-1)*Δt, the value range of i is [1, n], n*|Δt|=k*T, T is the period of the light signal, n is an integer greater than 1, k is an integer, and Δt is a preset time length;

[0090] performing surface fitting on the n initial depth images to obtain n fitted depth images;

[0091] calculating a field phase correction coefficient matrix according to the n fitted depth images after mean value processing, the field phase correction coefficient matrix being used to correct the field phase error of the depth image obtained by the shooting device through the TOF ranging method.

[0092] In one possible implementation, the initial depth image obtained in the i th time is the i th initial depth image, and the n initial depth images are obtained by:

[0093] obtaining n measured depth images actually measured by the shooting device through the TOF ranging method n times, the i th measured depth image in the n measured depth images being represented by a pixel matrix Di(x, y)';

[0094] The sum of each pixel value in the pixel matrix Di(x, y)' and c*[(1-i)*Δt] / 2 is calculated to obtain the ith initial depth image Di(x, y), wherein c*[(1-i)*Δt] / 2 is a distance difference caused by the delay of the starting time of the light source from the starting time of the sensor by (i-1)*Δt.

[0095] In one possible implementation, the field phase correction coefficient matrix is calculated according to the mean value processing of the n fitted depth images, including:

[0096] The pixel matrix of the n fitted depth images is averaged to obtain a fitted average pixel matrix.

[0097] The minimum value of the pixel values in the fitted average pixel matrix is extracted, and the ratio of the minimum value to each pixel value in the fitted average pixel matrix is calculated to obtain a ratio matrix, wherein the ratio matrix is the field phase correction coefficient matrix.

[0098] In one possible implementation, after the n initial depth images are respectively fitted to obtain the n fitted depth images, the method further includes:

[0099] The pixel matrix of the n initial depth images is averaged to obtain an initial average pixel matrix.

[0100] The pixel matrix of the n fitted depth images is averaged to obtain a fitted average pixel matrix.

[0101] The difference between the fitted average pixel matrix and the initial average pixel matrix is calculated to obtain a difference matrix, wherein the difference matrix is a fixed pattern noise (FPN) matrix, and each value in the FPN matrix includes a fixed noise of a pixel with the same subscript in the depth image caused by hardware.

[0102] In a seventh aspect, the present application provides a device including a processor and a communication interface, and the device is configured to execute the method of any one of the first aspect.

[0103] In one possible implementation, the device is a chip or a system on a chip (SoC).

[0104] In one possible implementation, the device is a chip or a system on a chip (SoC).

[0105] In a ninth aspect, the present application provides a computer readable storage medium storing a computer program, which is executed by a processor to implement the method of any one of the first aspect.

[0106] In a tenth aspect, the present application provides a computer readable storage medium storing a computer program, which is executed by a processor to implement the method of any one of the second aspect.

[0107] In an eleventh aspect, the present application provides a computer program product, when the computer program product is read and executed by a computer, the method of any one of the first aspect will be executed.

[0108] In a twelfth aspect, the present application provides a computer program product, when the computer program product is read and executed by a computer, the method of any one of the second aspect will be executed.

[0109] In a thirteenth aspect, the present application provides a computer program, when the computer program is executed on a computer, the computer will implement the method of any one of the first aspect.

[0110] In a fourteenth aspect, the present application provides a computer program, when the computer program is executed on a computer, the computer will implement the method of any one of the second aspect.

[0111] In summary, the depth nonlinear error is a nonlinear error caused by the fact that the waveform of the light signal is not a standard waveform (for example, not a standard sine wave, etc.), and the error amount of the nonlinear error is similar to that of a sine wave, which oscillates with distance changes, and the error amount is positive and negative. Therefore, in the present application, by sampling multiple times in the case of delaying the start of the light source of the shooting device relative to the sensor and performing mean processing according to the n initial depth images, the positive and negative error amounts in the nonlinear error can be offset, thereby reducing the depth nonlinear error introduced in the measurement process. At the same time, the field phase correction coefficient matrix obtained based on the sampling and mean processing operations can correct the depth value of each pixel in the depth image, reduce the error caused by the field phase, and greatly reduce the distortion rate of the final obtained depth image, thereby improving the quality of the depth image. BRIEF DESCRIPTION OF DRAWINGS

[0112] Figure 1 Fig. 1 shows a scene schematic diagram to which a depth image processing method provided by an embodiment of the present application is applicable;

[0113] Figure 2 Fig. 3 shows a training flowchart of a depth image correction amount provided by an embodiment of the present application;

[0114] Fig. 4 shows a depth image correction method provided by an embodiment of the present application;Figure 3 Fig. 1 shows a schematic diagram of a shooting scene according to an embodiment of the present application;

[0115] Figure 4 Fig. 2 shows a schematic diagram of a depth non-linear error according to an embodiment of the present application;

[0116] Figure 5 Fig. 3 shows a flowchart of a depth image processing method according to an embodiment of the present application;

[0117] Figure 6 Fig. 4 shows a flowchart of another depth image processing method according to an embodiment of the present application;

[0118] Figure 7 Fig. 5 shows a logical structure diagram of a shooting device according to an embodiment of the present application;

[0119] Figure 8 Fig. 6 shows a logical structure diagram of a computing device according to an embodiment of the present application;

[0120] Figure 9 Fig. 7 shows a hardware structure diagram of a shooting device according to an embodiment of the present application;

[0121] Figure 10 Fig. 8 shows a hardware structure diagram of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0122] Embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0123] Figure 1 Fig. 1 shows a schematic diagram of a shooting scene according to an embodiment of the present application. The scene includes a shooting device 101 and a shooting object 102. The shooting device 101 includes a light source 1011, a controller 1012, a sensor 1013 and a processor 1014. The light source 1011, the controller 1012, the sensor 1013 and the processor 1014 can be connected to each other through lines and / or interfaces.

[0124] The shooting device 101 is configured to acquire a depth image of the shooting object 102 by imaging the shooting object 102 through a time ranging method. The time ranging method can include, for example, a time of flight (TOF) ranging method and a laser radar ranging method.

[0125] The process of acquiring a depth image is illustrated using Time-of-Flight (TOF) ranging as an example. Specifically, the controller 1012 in the imaging device 101 sends a start signal to the light source 1011 and the sensor 1013 to activate them. Then, the light source 1011 emits a light signal towards the object 102. After reaching the object 102, the light signal is reflected back, and the imaging device 101 receives the reflected light signal through the sensor 1013. The imaging device 101 can then calculate the time difference t between any point on the imaging device 101 and the object 102 based on the phase difference between the received reflected light signal and the transmitted light signal. 1j The value of j ranges from [1, m], where m is the number of points sampled by the imaging device 101 from the object 102. Corresponding to the depth image of the object 102, m is the number of pixels in the pixel matrix of that depth image. Based on the time difference t... 1j The distance D1 between the imaging device 101 and the arbitrary point is calculated using the speed of light propagation c. j D1 j =c*t 1j / 2. Where t 1j =φ j / (2πf), where f is the frequency of the optical signal, φ j =arctan[(Q 3j -Q 1j ) / (Q 0j -Q 2j Q0, Q1, Q2, and Q3 are four images acquired by the sensor with time-domain phase differences of 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively. 0j Q 1j Q 2j and Q 3j These are the pixel values ​​of the pixels corresponding to any given point in the four images Q0, Q1, Q2, and Q3, respectively.

[0126] Alternatively, the imaging device 101 can record the moment the light signal is emitted and the moment the reflected light signal is received. By calculating the difference between these two moments, the time difference t between the light signal and any point on the imaging device 101 and the subject 102 can be calculated. 2j Then, based on that time difference t 2j The distance D between the imaging device 101 and the arbitrary point is calculated from the speed of light propagation c. 2j :D 2j =c*t 2j / 2.

[0127] Through either of the two ways, the shooting device 101 can obtain the depth value of each pixel point in the depth image of the shooting object 102, which is the value of the distance between the shooting device 101 and a certain point in the shooting object 102 corresponding to the pixel point, and can also be referred to as the pixel value of the pixel point. After obtaining the pixel value of each pixel point in the depth image, the pixel matrix of the depth image is obtained, which is the depth image of the shooting object 102 for the device.

[0128] It should be noted that in the above two ways of obtaining the depth image, the calculation of the depth value based on the measurement data measured by the TOF ranging method can be performed in the above-mentioned shooting device 101, or the shooting device 101 can send the measurement data to other devices such as a server in the cloud for calculation of the depth value. In addition, the controller 1012 can also be used for delay control, which adjusts the time difference between the start of the light source 1011 and the sensor 1013, and can be used for various calibrations. The controller 1012 can be a control circuit in an integrated chip, which principle is to generate a delay by using an RC circuit, for example, the controller 1012 can make the light source 1011 start later or start earlier than the sensor 1013 through the RC circuit. Alternatively, the controller can also be a control unit realized by software, which adjusts the time difference between the start of the light source 1011 and the sensor 1013 through software logic.

[0129] It should be noted that in this application, the start of the light source 1011 refers to the start of the light source 1011 to emit light signals to the shooting object 102, and the start of the sensor 1013 refers to the start of the sensor 1013 to be able to receive the reflected light signals of the shooting object 102. The light source 1011 starts to emit light signals later than the sensor 1013 starts to receive reflected light signals, which means that the light source 1011 starts to emit light signals at a time later than the sensor 1013 starts to receive reflected light signals. The light source 1011 starts to emit light signals earlier than the sensor 1013 starts to receive reflected light signals, which means that the light source 1011 starts to emit light signals at a time earlier than the sensor 1013 starts to receive reflected light signals. The laser ranging method can obtain the three-dimensional information of the scene by laser scanning. The basic principle is to emit laser to space at a certain time interval, and record the signal of each scanning point from the laser radar to the object (such as the shooting object 102) in the measured scene, and then the time interval of the reflected light from the object back to the laser radar, and the distance between the object surface and the laser radar is calculated, so that the depth image of the object in the measured scene can be obtained.

[0130] The photographing device 101 can be various types of cameras, mobile phones, tablets, computers, various types of camera heads, laser radars, and the like. The light source 1011 can be a light emitting diode (LED) or a laser, which can include a vertical-cavity surface-emitting laser (VCSEL) or a laser diode, and the like. The controller 1012 can be a control circuit or a control chip, and the like. The sensor 1013 can be a light sensor composed of a photosensitive element or an image sensor or an image sensor, and the like.

[0131] The processor 1014 can include one or more processors, for example, the processor 1014 can include one or more central processing units (CPUs) and one or more graphics processing units (GPUs). When the processor 1014 includes multiple processors, the multiple processors can be integrated on the same chip or can each be an independent chip.

[0132] In the embodiments of the present application, the CPU can be used to control the above-mentioned controller 1012, light source 1011 and sensor 1013 to complete the implementation of the above-mentioned time-of-flight method, thereby obtaining the measured data. The GPU can be used to calculate the depth image according to the measured data. The calculations can include one or more of the calculation of the field phase correction coefficient matrix, the calculation of the FPN matrix, the calculation of the field phase error of the pixel value of the corrected depth image, and the calculation of the fixed pattern noise of the pixel value of the depth image. The specific calculation process can be referred to in the following description, which is not described in detail here.

[0133] Of course, in another possible implementation, the processor 1014 can include one or more central processing units (CPUs), and the above-mentioned various calculations can be completed by the CPU.

[0134] The scene to which the depth image processing method provided in the embodiments of the present application is applied is not limited to the above-mentioned Figure 1 The scene to which the depth image processing method provided in the embodiments of the present application is applied is not limited to the above-mentioned

[0135] In specific embodiments, when the shooting device acquires the depth image of the shooting object, the obtained depth image is distorted due to errors caused by temperature offset, depth nonlinearity error, global time offset error, fixed pattern noise (FPN) and field of view phase error. The field of view phase error is an error caused by an angle deviation of the field of view, and the fixed noise is a fixed error generated in each pixel of the depth image caused by hardware.

[0136] In order to reduce the distortion rate of the depth image and acquire a high-quality depth image, the present application provides a depth image processing method, which can correct the field of view phase error of the depth image through a field of view phase correction coefficient matrix and / or reduce the fixed pattern noise through an FPN matrix, so as to greatly reduce the distortion rate of the obtained depth image and improve the quality of the depth image.

[0137] In the embodiments of the present application, the field of view phase correction coefficient matrix and / or the FPN matrix can be obtained by training in advance. The field of view phase correction coefficient matrix and / or the FPN matrix are obtained after the depth nonlinearity error is eliminated, so as to further reduce the distortion rate of the depth image.

[0138] Firstly, the training process of obtaining the field of view phase correction coefficient matrix and the FPN matrix is introduced below. Referring to FIG. 1, Figure 2 , the training process can include but is not limited to the following steps:

[0139] S201, a shooting device and a shooting object are built.

[0140] In the training process, the shooting object can be a flat plane, that is, a flat plane is used as a calibration reference for training. Referring to FIG. 2, Figure 3 , Figure 3 , a scene diagram for training is shown. In Figure 3 , the plane 301 is within the range of light signals emitted by the light source 302 of the shooting device, so that each point on the plane can be sampled. Moreover, the lens optical axis of the shooting device is perpendicular to the plane 301. Figure 3 The point A in the plane 301 shown in FIG. 3 can be the point closest to the lens, and the point A can be any point in the plane 301, which is not limited to the center point of the plane 301. After the plane and the shooting device are built, shooting can be started.

[0141] S202, a depth image of the shooting object is measured and acquired based on the time-of-flight method.

[0142] In the training process of the embodiments of the present application, whether the TOF ranging method or the laser radar ranging method or other ranging methods are used, a plurality of depth images of the shooting object are acquired by multiple measurements, and a depth image of the shooting object can be obtained each time.

[0143] It should be noted that after the measurement data is obtained through multiple measurements, subsequent various calculations can be performed on the shooting device according to the measurement data, or the shooting device can send the measurement data to other devices such as a cloud server and the like to perform subsequent various calculations, or the shooting device and the other device such as the cloud server can each complete a part of the calculation and the like. These calculations can include depth value calculation, surface fitting, averaging, ratio calculation and the like based on the measurement data, and the calculation process will be described below, which will not be described in detail here. In order to facilitate description, the device performing the calculation operation will be referred to as a calculation device, which can be a shooting device or a cloud server and the like.

[0144] In specific embodiments, n measurements can be performed, and the starting time of the light source in the i-th measurement is delayed from the starting time of the sensor by (i-1)*Δt, the value of i is in the range of [1, n], n*︱Δt︱=k*T, T is the period of the light signal emitted by the light source, n is an integer greater than 1, k is an integer, and Δt is a preset time length.

[0145] The Δt can be positive or negative. When the Δt is positive, then the above-mentioned starting time of the light source is delayed from the starting time of the sensor by (i-1)*Δt means that the sensor starts (i-1)*Δt earlier than the light source; when the Δt is negative, then the above-mentioned starting time of the light source is delayed from the starting time of the sensor by (i-1)*Δt means that the light source starts (i-1)*(-Δt) earlier than the sensor. The Δt can be any value other than 0. The light source and the sensor are the light source and the sensor in the shooting device, for example, they can be the light source 1011 and the sensor 1013 shown in FIG. 10. Figure 1

[0146] The starting time of the light source refers to the time when the light source starts to emit a light signal. Specifically, the starting time of the light source can be the time when the light source emits a light signal to the shooting object, for example, the plane 301 shown in FIG. 10. Figure 3

[0147] The starting time of the sensor refers to the time when the sensor starts to be able to receive a reflected light signal. Specifically, the time when the sensor starts to be able to receive a reflected light signal can be the time when the sensor starts to be able to receive a light signal reflected by the shooting object, for example, the plane 301 shown in FIG. 10. The reflected light signal is a light signal emitted by the light source to the shooting object, for example, the plane 301 shown in FIG. 10, and then reflected back to the shooting device, and then the shooting device receives the reflected light signal through the sensor. Figure 3 Figure 3 ​​​​

[0148] In order to facilitate the understanding of the start time of the light source and the start time of the sensor, the following is exemplified. Exemplarily, at the first second, the light source starts to emit the light signal, that is, the start time of the light source is the first second; at the second second, the sensor starts to be able to receive the reflected light signal, that is, the start time of the sensor is the second second; at the third second, the sensor starts to receive the reflected light signal. Here, the sensor can start to receive the reflected light signal at the second second, but there is no reflected light signal transmitted to the sensor, so the sensor does not receive the reflected light signal at the second second. At the third second, the reflected light signal starts to be transmitted to the sensor, so the sensor starts to receive the reflected light signal at the third second. This is only exemplarily introduced and explained, and the specific start time of the light source, the start time of the sensor and the time at which the sensor starts to receive the reflected light signal are determined according to actual conditions, and the present scheme does not limit this.

[0149] The start time of the light source of the i-th time in the n times is delayed by (i-1)*Δt than the start time of the sensor, which indicates that for the start time of the light source being delayed than the start time of the sensor, each measurement is delayed by Δt than the previous measurement.

[0150] Exemplarily, assuming that n is 4, then:

[0151] At the first measurement, the start time of the light source is delayed by 0 than the start time of the sensor, that is, the light source and the sensor are started at the same time, which means that the time at which the light source starts to emit the light signal and the time at which the sensor starts to be able to receive the reflected light signal are the same;

[0152] At the second measurement, the start time of the light source is delayed by Δt than the start time of the sensor;

[0153] At the third measurement, the start time of the light source is delayed by 2*Δt than the start time of the sensor;

[0154] At the fourth measurement, the start time of the light source is delayed by 3*Δt than the start time of the sensor.

[0155] This is only exemplarily explained, and the specific value of n is determined according to actual needs, and the present scheme does not limit this.

[0156] The n*︱Δt︱=k*T indicates that the total amount of time of the n times of delay is an integer multiple of the period of the light signal emitted by the light source. Specifically, the start times of the light source and the sensor can be controlled by a controller, which can be the controller 1012 described in the Figure 1 . That is, the controller can generate a delay by using an RC circuit or by using software, so as to realize that the light source is delayed to start (i-1)*Δt than the sensor.

[0157] The light source is delayed than the sensor refers to the time when the light source starts to emit the light signal is later than the time when the sensor starts to receive the reflected light signal. The light source is earlier than the sensor refers to the time when the light source starts to emit the light signal is earlier than the time when the sensor starts to receive the reflected light signal.

[0158] The n measurements are taken and the delay At is set such that n*|At| = k*T is to reduce the depth nonlinearity error. When the emission pattern of the active light source and the activation pattern of the sensor are not perfect sinusoidal waves, the measured depth values will have nonlinearity error. The nonlinearity error has different error values at different distances, and the relationship between the error amount and the depth is similar to that of a sinusoidal wave, which oscillates with distance and has periodic repetition. As shown in Figure 4 The depth nonlinearity error example at a frequency of 100Mhz is shown, where the x-axis is the depth value, and the y-axis is the nonlinearity error amount corresponding to the depth. It can be observed that this nonlinearity error has periodic oscillation and two characteristics of 0 and (the total error in a complete period is 0). How to use the periodic oscillation and 0 and to reduce the depth nonlinearity error will be described below, which will not be described here.

[0159] For example, in the first measurement, the light source and the sensor in the shooting device are started at the same time, that is, the light source starts to emit the light signal to the shooting object at the same time as the sensor starts to sense the light signal reflected by the shooting object. Then, the computing device calculates the depth value of each pixel point of the depth image of the shooting object based on the information of the emitted light signal and the information of the reflected light signal according to the calculation method described above in the description of Figure 1 The depth image D1(x, y) of the shooting object can be referred to as the first initial depth image. Wherein, the D1(x, y) is the pixel matrix of the first initial depth image, and the D1(x, y) is the first initial depth image for the device. Each pixel value in the pixel matrix D1(x, y) is the distance value, i.e. the depth value, between the shooting device and a certain point in the shooting object corresponding to each pixel point.

[0160] In the above D1(x, y), x represents the number of rows of the pixel matrix, and y represents the number of columns of the pixel matrix, that is, D1(x, y) is a pixel matrix of x rows and y columns. The x and y in the matrix below also represent the number of rows and columns of the matrix respectively. It will not be described below.

[0161] In the i-th (i is greater than 1) measurement, the starting time of the light source in the shooting device is delayed than the starting time of the sensor by (i-1)*At, in which case the light source emits the light signal to the shooting object, and then the sensor senses the light signal reflected by the shooting object. The computing device calculates the depth value of each pixel point of the depth image of the shooting object based on the information of the emitted light signal and the information of the reflected light signal according to the calculation method described above in the description ofFigure 1 The calculation method introduced in the description of the application calculates the depth value of each pixel point of the depth image of the photographed object, thereby obtaining the depth image Di(x, y)' of the photographed object. Since the depth image Di(x, y)' is directly calculated according to the measured data, the depth image Di(x, y)' can be referred to as the ith measured depth image. Di(x, y)' is the pixel matrix of the ith measured depth image, and Di(x, y)' is the ith measured depth image for the device. Each pixel value in the pixel matrix Di(x, y)' is the value of the distance between the photographed device and a certain point in the photographed object corresponding to each pixel point, that is, the depth value.

[0162] However, since the light source is started with a delay of (i-1)*Δt, it is equivalent to that the sensor receives the reflected light signal with a delay of (i-1)*Δt, which causes the calculated transmission time of the light signal to increase by (i-1)*Δt, so that the depth value calculated according to the transmission time increases by c*[(i-1)*Δt] / 2. Therefore, the depth value of each pixel in the ith measured depth image should be reduced by c*[(i-1)*Δt] / 2 to be the true depth value, or increased by c*[(1-i)*Δt] / 2 to be the true depth value. That is, c*[(1-i)*Δt] / 2 or c*[(i-1)*Δt] / 2 is the distance difference caused by the delay of (i-1)*Δt of the starting time of the light source compared with the sensor.

[0163] Then, the ith initial depth image finally calculated based on the ith measurement can be represented by a pixel matrix Di(x, y). For the device, Di(x, y) is the ith initial depth image, and each pixel value in the pixel matrix Di(x, y) is obtained by adding c*[(1-i)*Δt] / 2 to each pixel value in the pixel matrix Di(x, y)'. Alternatively, the pixel matrix Di(x, y) is obtained by subtracting c*[(i-1)*Δt] / 2 from each pixel value in the pixel matrix Di(x, y)'.

[0164] For the above-mentioned first measurement, since the sensor and the light source are started at the same time without time delay, the first measured depth image obtained based on the measurement data is the first initial depth image, and the distance difference caused by the time delay does not need to be considered. It should be noted that the sizes of the n initial depth images D1(x, y)~Dn(x, y) calculated according to the above steps are the same, that is, D1(x, y)~Dn(x, y) are n matrices with equal number of rows and equal number of columns.

[0165] S203, surface fitting is performed on the n initial depth images obtained based on the above measurement.

[0166] After the computing device obtains the n initial depth images D1(x, y)~Dn(x, y) described above, the n initial depth images can be respectively fitted with a surface to obtain n fitted depth images, and the pixel matrix of the n fitted depth images can be represented as Df1(x, y)~Dfn(x, y), and the fitted depth image obtained by fitting the i initial depth image is the i fitted depth image Df i(x, y). For the device, the Df i(x, y) is the i fitted depth image.

[0167] It should be noted that the n fitted depth images Df1(x, y)~Dfn(x, y) calculated according to the above steps are all of the same size, that is, the Df1(x, y)~Dfn(x, y) are n matrices with equal rows and equal columns. In addition, the size of the n fitted depth images is the same as that of the n initial depth images described above, that is, the D1(x, y)~Dn(x, y) and the Df1(x, y)~Dfn(x, y) are 2*n matrices with equal rows and equal columns.

[0168] Optionally, the surface fitting method described above can be a polynomial surface fitting method of least squares or a surface fitting method of other adaptive functions. Specifically, after obtaining the pixel matrix of the n initial depth images, the approximate function type of the image can be roughly judged according to the pixel matrix, for example, the scatter plot image corresponding to the pixel matrix can be first drawn or simulated by using drawing software or simulation software, and then the approximate function type of the scatter plot image can be judged according to experience. After the approximate function is determined, the function can be used to fit the n initial depth images respectively to obtain n fitted depth images.

[0169] Exemplarily, assuming that the n initial depth images can be fitted with a polynomial surface fitting method of least squares, then the fitted polynomial can be formula (1): surface(x, y) = p 00 +p 10 *x+p 01 *y+p 20 *x^2+p 11 *x*y+p 02 *y^2, or the fitted polynomial can be formula (2): surface(x, y) = p 00 +p 10 *x+p 01 *y+p 20 *x^2+p 11 *x*y+p 02 *y^2+p 30 *x^3+p 21 *x^2*y+p 12 *x*y^2+p 03y3+ p 40 x4+ p 31 x3y+ p 22 x2y2+ p 13 xy3+ p 04 y4. Wherein, the surface(x, y) is the above Dfi(x, y), and p is a coefficient of the polynomial. Through the fitting of the depth image, the depth value of any pixel point can be calculated according to the values of x and y.

[0170] In formula (1), the highest degree of the monomial is 2, therefore, the surface fitting by using formula (1) can be called quadratic polynomial surface fitting. In formula (2), the highest degree of the monomial is 4, therefore, the surface fitting by using formula (2) can be called quartic polynomial surface fitting. The highest degree of the monomial in the polynomial needs to be determined according to actual needs, if too small, the fitting surface obtained is quite different from the actual surface, if too large, over-fitting phenomenon will occur.

[0171] Here, only the polynomial is exemplarily introduced for surface fitting, in specific embodiments, other suitable functions can also be used for surface fitting, which is not limited in the present solution.

[0172] S204, calculating the fitting average pixel matrix according to the n fitting depth images calculated in S203.

[0173] After obtaining the n fitting depth images Df1(x, y)~Dfn(x, y), the pixel matrices of the n fitting depth images can be averaged to obtain a fitting average depth image. The fitting average depth image can be represented by a fitting average pixel matrix Dfa(x, y), which is the fitting average depth image for the device. Specifically, the pixel values with the same subscript in the pixel matrices Df1(x, y)~Dfn(x, y) of the n fitting depth images can be averaged respectively to obtain the fitting average pixel matrix Dfa(x, y). Exemplarily, the specific calculation formula can be as follows:

[0174]

[0175] In the embodiments of the present application, the same subscript means the same number of rows and the same number of columns in the matrix, and then the pixel values with the same subscript mean the pixel values at the positions with the same number of rows and the same number of columns in the matrix.

[0176] According to the foregoing description, the depth nonlinearity error has the characteristics of periodic oscillation and 0 and two, the measurement sampling process from Df1(x, y) to Dfn(x, y) happens to contain a complete period of oscillation, and therefore, the average of the n fitting depth images Df1(x, y)~Dfn(x, y) Dfa(x, y) can make the nonlinearity periodic oscillation offset each other, thereby reducing the depth nonlinearity error.

[0177] S205, calculating the field phase correction coefficient matrix according to the calculated fitting average pixel matrix.

[0178] After the fitting average pixel matrix Dfa(x, y) is calculated, the minimum value d0 in the Dfa(x, y) is extracted, and the point corresponding to the minimum value d0 can be a point in the area closest to the lens of the shooting device in the shooting object. For example, it can be a point in the A point or the area near the A point in the above Figure 3 Then, the ratio of the minimum value d0 to each pixel point in the fitting average pixel matrix Dfa(x, y) is calculated to obtain a ratio matrix S1(x, y). The ratio matrix S1(x, y) is the above-mentioned field phase correction coefficient matrix.

[0179] Since the field phase correction coefficient matrix S1(x, y) is the ratio of the minimum value d0 to each pixel point in the fitting average pixel matrix Dfa(x, y), each value in the field phase correction coefficient matrix S1(x, y) is less than or equal to 1.

[0180] S206, calculating the FPN matrix according to the pixel matrix of the n initial depth images and the fitting average pixel matrix.

[0181] First, the pixel matrix of the n initial depth images is averaged to obtain an initial average depth image, which can be represented by an initial average pixel matrix Da(x, y). For the device, the initial average pixel matrix Da(x, y) is the initial average depth image. Specifically, the pixel values with the same subscript in the pixel matrix D1(x, y)~Dn(x, y) of the n initial depth images can be averaged respectively to obtain the initial average pixel matrix Da(x, y). Exemplarily, the specific calculation formula can be as follows:

[0182]

[0183] Then, the difference between the fitting average pixel matrix Dfa(x, y) and the initial average pixel matrix Da(x, y) is calculated to obtain a difference matrix S2(x, y), and the specific calculation formula can be as follows:

[0184] S2(x, y) = Dfa(x, y) - Da(x, y).

[0185] The difference matrix S2(x, y) is the FPN matrix.

[0186] It should be noted that the execution order of S205 and S206 is not sequential, that is, S205 can be executed first, and then S206 can be executed, or S206 can be executed first, and then S205 can be executed, or S205 and S206 can be executed simultaneously.

[0187] In one possible implementation, the field phase correction coefficient matrix can also be calculated by the following method:

[0188] The ratio matrix S1(x, y)' is obtained by calculating the ratio of each pixel point in the fitted average pixel matrix Dfa(x, y) to the minimum value d0. The ratio matrix S1(x, y)' can also be a field phase correction coefficient matrix.

[0189] Since the field phase correction coefficient matrix S1(x, y)' is obtained by calculating the ratio of each pixel point in the fitted average pixel matrix Dfa(x, y) to the minimum value d0, each value in the field phase correction coefficient matrix S1(x, y)' is greater than or equal to 1.

[0190] In one possible implementation, the FPN matrix can also be calculated by the following method:

[0191] The difference matrix S2(x, y)' is obtained by calculating the difference between the initial average pixel matrix Da(x, y) and the fitted average pixel matrix Dfa(x, y). The specific calculation formula can be as follows:

[0192] S2(x, y) = Da(x, y) - Dfa(x, y)

[0193] The difference matrix S2(x, y)' can also be an FPN matrix.

[0194] It should be noted that in specific embodiments, the field phase correction coefficient matrix and the FPN matrix can be trained to obtain one of the matrices. For example, if you want to correct the field phase error in the depth image, you can train the field phase correction coefficient matrix; if you want to reduce the fixed pattern noise in the depth image, you can train the FPN matrix. Of course, if you want to correct the field phase error in the depth image and reduce the fixed pattern noise in the depth image at the same time, you can train the field phase correction coefficient matrix and the FPN matrix.

[0195] In one possible implementation, when calculating the field phase correction coefficient matrix, the n initial depth images can not be fitted with a curved surface, but the pixel matrix of the n initial depth images can be averaged to obtain an initial average depth image Da(x, y)', and then the minimum value d0' in Da(x, y)' is extracted, which can be a point in the area closest to the lens of the shooting device in the shooting object. For example, it can be a point in the area of point A or around point A in the above Figure 3 Then, the ratio of the minimum value d0' to each pixel in the initial average pixel matrix Da(x, y)' is calculated to obtain a ratio matrix S1(x, y)". The ratio matrix S1(x, y)" is the field phase correction coefficient matrix. Alternatively, the ratio of each pixel in the initial average pixel matrix Da(x, y)' to the minimum value d0' is calculated to obtain a ratio matrix S1(x, y)"', which can also be a field phase correction coefficient matrix.

[0196] The above Figure 2 In the training method, the two characteristics of periodic oscillation and zero sum (the total error in a complete cycle is zero) of the depth nonlinear error are used, the light source sensor of the shooting device is delayed multiple times, the total time of the multiple delays is an integer multiple of the period of the sampling light signal, and then the pixel matrix of the depth images obtained by the multiple delays is averaged to reduce the depth nonlinear error, thereby improving the accuracy of the obtained field phase correction coefficient matrix. Therefore, the field phase correction coefficient matrix of the depth image obtained by the embodiment of the application can correct the field phase error of each pixel value in the depth image obtained by the shooting device, thereby reducing the distortion rate of the depth image and improving the quality of the depth image.

[0197] After obtaining the field phase correction coefficient matrix and / or the FPN matrix, the data of the obtained matrix can be stored in the shooting device for correcting the error of each pixel value in the depth image obtained by the shooting device. Optionally, the shooting device can be the same device as the shooting device used to train the field phase correction coefficient matrix and the FPN matrix; or the shooting device can be the same model as the shooting device used to train the field phase correction coefficient matrix and the FPN matrix, or a shooting device with the same or similar hardware performance; or the shooting device can be any device that can obtain a depth image, etc.

[0198] Based on the above description, a depth image processing method provided by an embodiment of the application is introduced below. The method includes the process of correcting the pixel error of a depth image by a shooting device based on the field phase correction coefficient matrix and / or the FPN matrix. Referring to Figure 5The method can include, but is not limited to, the following steps:

[0199] S501, the shooting device acquires a first depth image of a first object.

[0200] The first object can be any object shot by the shooting device, and can be a planar object, a three-dimensional object, or a spatial object, etc. Alternatively, the first depth image can also be the above-mentioned Figure 2 The first initial depth image D1(x, y) in the training process, etc. The present scheme does not limit the specific object to be shot.

[0201] Specifically, the controller in the shooting device can send a signal to simultaneously start the light source and the sensor to measure data for calculating the first depth image. For specific calculation of the first depth image from the measured data, please refer to the related description in the above introduction of the first depth image, which will not be repeated here. Figure 1

[0202] S502, the shooting device corrects the depth value of the pixel in the first depth image by a field phase correction coefficient matrix to obtain a second depth image, wherein the field phase correction coefficient matrix is a matrix for correcting the field phase error obtained by preprocessing n initial depth images, the n initial depth images are depth images obtained in a first case, the first case includes that the starting time of the light source of the shooting device and the starting time of the sensor of the shooting device are different time points, the preprocessing includes mean processing made according to the n initial depth images, and n is an integer greater than 1.

[0203] In the specific implementation process, the above-mentioned first case also includes the case that the starting time of the light source of the shooting device and the starting time of the sensor of the shooting device are the same time point. The above-mentioned mean processing made according to the n initial depth images can include respectively averaging the pixel values of the same subscript in the pixel matrix of the n initial depth images; or, including first performing surface fitting on the n initial depth images to obtain n fitting depth images, and then respectively averaging the pixel values of the same subscript in the pixel matrix of the n fitting depth images. The preprocessing also includes surface fitting, ratio taking, etc. The specific implementation process of the mean processing, surface fitting, and ratio processing, etc. can refer to the description of the training process in the above-mentioned Figure 2

[0204] In specific embodiments, the shooting device has already stored data of the field phase correction coefficient matrix, which can be the above-mentioned Figure 2 ​​The field phase correction coefficient matrix S1(x, y) is obtained in the training process. The size of the first depth image can be the same as that of the field phase correction coefficient matrix S1(x, y). For example, assuming that the size of the field phase correction coefficient matrix S1(x, y) is 1024*1024, the size of the pixel matrix of the first depth image is also 1024*1024.

[0205] The field phase correction coefficient matrix S1(x, y) is obtained according to the n fitting depth images. Specifically, the field phase correction coefficient matrix is a ratio matrix obtained by taking the ratio of the minimum value of each pixel value in the fitting average pixel matrix to each pixel value in the fitting average pixel matrix. The fitting average pixel matrix is obtained by averaging the pixel values with the same subscript in the pixel matrix of the n fitting depth images.

[0206] The n fitting depth images are obtained by surface fitting the n initial depth images. The n initial depth images are depth images of the second object obtained by the time-of-flight (TOF) ranging method n times based on the training camera. The second object is the training plane. The starting time of the light source in the i-th time is delayed by (i-1)*Δt from the starting time of the sensor. The value of i ranges from 1 to n. n*|Δt|=k*T, where T is the period of the light signal, k is an integer, and Δt is a preset time length.

[0207] In addition, the initial depth image obtained in the i-th time is the i-th initial depth image. The i-th initial depth image is represented by a pixel matrix Di(x, y). Each pixel value in the pixel matrix Di(x, y) is obtained by taking the sum of each pixel value in the pixel matrix Di(x, y)' and c*[(1-i)*Δt] / 2. Di(x, y)' is the pixel matrix of the depth image obtained by the i-th measurement. c*[(1-i)*Δt] / 2 is the distance difference caused by the delay of the starting time of the light source from the starting time of the sensor by (i-1)*Δt.

[0208] The process of training to obtain the field phase correction coefficient matrix S1(x, y) can be referred to the corresponding description in the above Figure 2 .

[0209] Then, the second depth image is obtained by correcting the depth value of the pixel in the first depth image by the field phase correction coefficient matrix. Specifically, the product matrix is obtained by calculating the product of the pixel matrix of the first depth image and the pixel value with the same subscript in the field phase correction coefficient matrix S1(x, y), and the product matrix is taken as the second depth image. The product matrix can be represented by DS(x, y), and the second depth image can be represented by DS(x, y).

[0210] The multiplication of elements of two matrices with the same subscript is called dot multiplication, and the symbol of dot multiplication is ".*". The pixel matrix of the first depth image can be denoted as A(x, y), and the product of the pixel matrix of the first depth image and the pixel value of the same subscript in the field phase correction coefficient matrix S1(x, y) is denoted as a product matrix, which can be expressed as follows:

[0211] DS(x, y) = A(x, y).* S1(x, y).

[0212] In a possible implementation, the field phase correction coefficient matrix in S502 can be S1(x, y)' described above. Then, the field phase correction coefficient matrix S1(x, y)' is obtained according to the n fitting depth images described above, and specifically, the field phase correction coefficient matrix is a ratio matrix obtained by respectively taking the ratio of each pixel value of the fitting average pixel matrix and the minimum value of the plurality of pixel values in the fitting average pixel matrix, and the fitting average pixel matrix is obtained by respectively averaging the pixel values of the same subscript in the pixel matrices of the n fitting depth images. The other descriptions are the same as those of the field phase correction coefficient matrix S1(x, y), which will not be described here.

[0213] Then, the second depth image is obtained by correcting the depth value of the pixel in the first depth image by the field phase correction coefficient matrix, and specifically, the ratio matrix obtained by respectively calculating the ratio of the pixel matrix of the first depth image and the pixel value of the same subscript in the field phase correction coefficient matrix S1(x, y)' is taken as the second depth image. Based on the training process and possible implementation thereof shown in the above Figure 2 It can be known from the training process and possible implementation thereof shown in the above that the S1(x, y)' and the value of the same subscript in the S1(x, y) described above are reciprocal of each other, so the ratio matrix and the product matrix DS(x, y) described above can be the same matrix, and the ratio matrix can be denoted as DS(x, y), that is, the second depth image can be denoted as DS(x, y).

[0214] In a possible implementation, the field phase correction coefficient matrix in S502 can be S1(x, y)". Then, the second depth image is obtained by correcting the depth value of the pixel in the first depth image by the field phase correction coefficient matrix, and specifically, the product matrix obtained by respectively calculating the product of the pixel matrix of the first depth image and the pixel value of the same subscript in the field phase correction coefficient matrix S1(x, y)" is taken as the second depth image. The product matrix can be denoted as DS(x, y)', that is, the second depth image can be denoted as DS(x, y)'. Then, the calculation process can be expressed as follows:

[0215] DS(x, y)' = A(x, y).* S1(x, y)".

[0216] In a possible implementation, the field of view phase correction coefficient matrix in S502 can be S1(x, y)’’’ described above. Then, the second depth image is obtained by correcting the depth values of the pixels in the first depth image by the field of view phase correction coefficient matrix, specifically: a ratio matrix is obtained by calculating the ratio of the pixel values of the pixel matrix of the first depth image and the pixel values of the same subscript in the field of view phase correction coefficient matrix S1(x, y)’’’ respectively, as the second depth image. Based on the training process and its possible implementation shown above, it can be known that the values of the same subscript in S1(x, y)’’’ and S1(x, y)’’ are reciprocal of each other, and therefore the ratio matrix and the product matrix DS(x, y)’ described above can be the same matrix, and the ratio matrix can be represented by DS(x, y)’, that is, the second depth image can be represented by DS(x, y)’. Figure 2

[0217] In the embodiments of the present application, by sampling multiple times in the case of delaying the start of the light source ratio sensor of the shooting device in the training process and the mean processing based on the n initial depth images, the positive and negative error amounts in the nonlinear error can be offset, so that the depth nonlinear error introduced in the measurement process can be reduced. Meanwhile, the field of view phase correction coefficient matrix obtained based on the sampling and mean processing and the like can correct the depth values of each pixel in the depth image, reduce the error caused by the field of view phase, and greatly reduce the distortion rate of the final obtained depth image, thereby improving the quality of the depth image.

[0218] In a possible implementation, referring to Figure 6 The depth image processing method provided by the embodiments of the present application can further include but is not limited to the following steps:

[0219] S503, the shooting device corrects the depth values of each pixel in the second depth image by the fixed pattern noise FPN matrix to obtain a third depth image.

[0220] In specific embodiments, the data of the FPN matrix has been stored in the shooting device, and the FPN matrix can be S2(x, y) obtained by the training process described above. Figure 2 The FPN matrix S2(x, y) obtained by the training process.

[0221] The size of the first depth image, the size of the FPN matrix and the size of the field of view phase correction coefficient matrix S1(x, y) are the same, for example, assuming that the size of the field of view phase correction coefficient matrix S1(x, y) is 1024*1024, then the size of the pixel matrix of the first depth image and the size of the FPN matrix are also 1024*1024.

[0222] ​That is, the FPN matrix is ​​the difference matrix S2(x,y) obtained by taking the difference between the fitted average pixel matrix and the initial average pixel matrix. The fitted average pixel matrix is ​​obtained by averaging the pixel values ​​with the same index in the pixel matrices of the n fitted depth images, and the initial average pixel matrix is ​​obtained by averaging the pixel values ​​with the same index in the pixel matrices of the n initial depth images. The specific process of obtaining the FPN matrix S2(x,y) during training can be found above. Figure 2 The corresponding descriptions are not repeated here.

[0223] Therefore, the third depth image is obtained by correcting the depth value of each pixel in the second depth image using the fixed-mode noise FPN matrix as follows: Given the matrix DS(x,y) calculated in S502, the sum of DS(x,y) and the FPN matrix S2(x,y) is calculated to obtain the sum matrix Dc(x,y), which serves as the third depth image. The specific calculation formula is as follows:

[0224] Dc(x,y)=DS(x,y)+S2(x,y).

[0225] Alternatively, if matrix DS(x,y)' is obtained from S502 above, the sum of matrix DS(x,y)' and FPN matrix S2(x,y) is calculated to obtain sum matrix Dc(x,y)' as the third depth image. The specific calculation formula is as follows:

[0226] Dc(x,y)'=DS(x,y)'+S2(x,y).

[0227] In one possible implementation, the FPN matrix in S503 can be S2(x,y)'. Then, the FPN matrix S2(x,y)' is the difference matrix obtained by taking the difference between the initial average pixel matrix Da(x,y) and the fitted average pixel matrix Dfa(x,y). The other descriptions are the same as those for the FPN matrix S1(x,y), and will not be repeated here.

[0228] Therefore, the method of correcting the depth value of each pixel in the second depth image using the fixed-mode noise FPN matrix to obtain the third depth image is as follows: Given the matrix DS(x,y) calculated in S502, the difference between DS(x,y) and the FPN matrix S2(x,y)' is calculated to obtain the difference matrix Dc(x,y)" as the third depth image. The specific calculation formula is as follows:

[0229] Dc(x,y)”=DS(x,y)-S2(x,y)′.

[0230] Based on the above Figure 2As can be seen from the training process and its possible implementation, the values ​​of the same index in S2(x,y)' and the above S2(x,y) are opposites of each other. Therefore, the difference matrix Dc(x,y)” and the above sum matrix Dc(x,y) can be the same matrix.

[0231] Alternatively, if matrix DS(x,y)' is obtained from S502 above, the difference between matrix DS(x,y)' and FPN matrix S2(x,y)' is calculated to obtain the difference matrix Dc(x,y)"' as the third depth image. The specific calculation formula is: Dc(x,y)'.

[0232] Dc(x,y)'''=DS(x,y)'-S2(x,y)'.

[0233] Similarly, based on the above Figure 2 As can be seen from the training process and its possible implementation, the values ​​of the same index in S2(x,y)' are opposites of each other. Therefore, the difference matrix Dc(x,y)”' and the sum matrix Dc(x,y)' can be the same matrix.

[0234] In the embodiments of this application, in addition to correcting the field-of-view phase error of the depth image, it can also correct the fixed-pattern noise of each pixel caused by the hardware of the shooting device.

[0235] The foregoing mainly describes the depth image processing method provided in the embodiments of this application. It is understood that each device, in order to achieve the corresponding functions, includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0236] This application embodiment can divide the device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0237] When dividing each function into modules according to its corresponding function.Figure 7 A possible logical structure diagram of a device is shown, which can be the device described above Figure 5 or Figure 6 the photographing device in the method. The photographing device 700 comprises an acquisition unit 701 and a correction unit 702. Wherein:

[0238] The acquisition unit 701 is configured to acquire a first depth image of a first object;

[0239] The correction unit 702 is configured to correct a depth value of a pixel in the first depth image by a field phase correction coefficient matrix to obtain a second depth image, wherein the field phase correction coefficient matrix is a matrix for correcting a field phase error obtained by preprocessing n initial depth images, the n initial depth images are depth images obtained in a first case, the first case includes that a starting moment of a light source of the photographing device and a starting moment of a sensor of the photographing device are different moments, the preprocessing includes mean processing made according to the n initial depth images, and n is an integer greater than 1.

[0240] In a possible implementation, the correction unit 702 is further configured to:

[0241] correct the second depth image by a fixed pattern noise (FPN) matrix to obtain a third depth image, wherein each value in the FPN matrix respectively comprises a fixed noise of a pixel in the first depth image with a same subscript as the each value caused by hardware.

[0242] In a possible implementation, the correction unit 702 is specifically configured to:

[0243] respectively calculate a product of a pixel matrix of the first depth image and a pixel value with a same subscript in the field phase correction coefficient matrix to obtain the second depth image.

[0244] In a possible implementation, the correction unit 702 is specifically configured to:

[0245] respectively calculate a product of a pixel matrix of the first depth image and a pixel value with a same subscript in the field phase correction coefficient matrix to obtain a product matrix;

[0246] calculate a sum of the product matrix and the FPN matrix to obtain the third depth image.

[0247] Figure 7 The specific operations and beneficial effects of each unit in the photographing device 700 shown can be referred to the descriptions of the photographing device 700 Figure 5 or Figure 6 and the method embodiments shown in possible implementations thereof, which will not be described herein again.

[0248] In the case of dividing each functional module according to each function, Figure 8 A possible logical structure diagram of a device is shown, which can be the above Figure 2 The computing device in the method. The computing device 800 comprises an acquisition unit 801, a curved surface fitting unit 802 and a calculation unit 803. Among them:

[0249] The acquisition unit 801 is used for acquiring n initial depth images, the n initial depth images being depth images of a second object obtained by a shooting device through time-of-flight (TOF) ranging method n times, wherein the starting time of the light source of the shooting device in the i-th time in the n times is delayed by (i-1)*Δt from the starting time of the sensor of the shooting device, the value range of i is [1, n], n*︱Δt︱=k*T, T is the period of the light signal, n is an integer greater than 1, k is an integer, and Δt is a preset time length;

[0250] The curved surface fitting unit 802 is used for respectively performing curved surface fitting on the n initial depth images to obtain n fitted depth images;

[0251] The calculation unit 803 is used for calculating a field phase correction coefficient matrix after mean value processing according to the n fitted depth images, the field phase correction coefficient matrix being used for correcting the field phase error of the depth image obtained by the shooting device through the TOF ranging method.

[0252] In one possible implementation, the initial depth image obtained in the i-th time is the i-th initial depth image; the acquisition unit 801 is specifically used for:

[0253] Acquiring n measurement depth images actually measured by the shooting device through time-of-flight (TOF) ranging method n times, the i-th measurement depth image in the n measurement depth images being represented by a pixel matrix Di(x, y)';

[0254] Respectively calculating the sum of each pixel value in the pixel matrix Di(x, y)' and c*[(1-i)*Δt] / 2 to obtain the i-th initial depth image Di(x, y), c*[(1-i)*Δt] / 2 being a distance difference caused by the starting time of the light source being delayed by (i-1)*Δt from the starting time of the sensor.

[0255] In one possible implementation, the calculation unit 803 is specifically used for:

[0256] Averaging the pixel values with the same subscript in the pixel matrices of the n fitted depth images respectively to obtain a fitted average pixel matrix;

[0257] extracting a minimum value of the pixel values in the fitted average pixel matrix, and calculating a ratio matrix by dividing the minimum value by each of the pixel values in the fitted average pixel matrix, the ratio matrix being the phase correction coefficient matrix of the field of view.

[0258] In one possible implementation, the calculation unit 803 is further configured to, after the curved surface fitting unit 802 performs curved surface fitting on the n initial depth images respectively to obtain n fitted depth images,

[0259] averaging the pixel values with the same subscript in the pixel matrices of the n initial depth images respectively to obtain an initial average pixel matrix;

[0260] averaging the pixel values with the same subscript in the pixel matrices of the n fitted depth images respectively to obtain a fitted average pixel matrix;

[0261] calculating a difference matrix by taking a difference between the fitted average pixel matrix and the initial average pixel matrix, the difference matrix being a fixed pattern noise (FPN) matrix, each value in the FPN matrix including a fixed noise of a pixel with the same subscript as the value in the depth image caused by hardware.

[0262] Figure 8 The specific operations and advantages of each unit in the illustrated computing device 800 can be found in the descriptions of the above-mentioned Figure 2 and the method embodiments in the possible implementations thereof, which will not be repeated here.

[0263] Figure 9 Fig. 8 shows a possible hardware structure of the device provided in the present application. The device can be the above-mentioned Figure 5 or Figure 6 photographing device in the method. The photographing device 900 includes a processor 901, a memory 902 and a communication interface 903. The processor 901, the communication interface 903 and the memory 902 can be connected with each other or connected with each other through a bus 904.

[0264] For example, the memory 902 is configured to store the computer programs and data of the photographing device 900. The memory 902 can include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a compact disc read-only memory (CD-ROM), etc. In the case of implementing the illustrated embodiments, and Figure 7 the processor 901 is configured to execute the computer programs stored in the memory 902. Figure 7In the case where each unit described in the embodiments is implemented by software, a software or program code required for the functions of the acquisition unit 701 and the correction unit 702 in the Figure 7

[0265] The communication interface 903 is configured to support the photographing device 900 to communicate, for example, to receive or send data or signals, etc.

[0266] The processor 901 can be a central processing unit, a graphics processing unit (GPU), a general purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. The processor can also be a combination of implementing computing functions, such as one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc. The processor 901 can be configured to read the program stored in the memory 902 and execute the operations described above. Figure 5 Figure 6 and the operations performed by the photographing device in the method described in the possible implementation.

[0267] Figure 10 Fig. 1 shows a possible hardware structure of the device provided in the present application. The device can be the computing device in the method described above. Figure 2 The computing device 1000 includes a processor 1001, a memory 1002, and a communication interface 1003. The processor 1001, the communication interface 1003, and the memory 1002 can be connected to each other or connected through a bus 1004.

[0268] The memory 1002 is configured to store computer programs and data of the computing device 1000. The memory 1002 can include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read only memory (EPROM), a compact disc read-only memory (CD-ROM), etc. In the case where the embodiment shown in Fig. 1 is implemented, the software or program code required for the functions of the acquisition unit 801, the curved surface fitting unit 802, and the calculation unit 803 in the Figure 8 Figure 8 In the case where each unit described in the embodiments is implemented by software, a software or program code required for the functions of the acquisition unit 701 and the correction unit 702 in the Figure 8

[0269] ​​​​The communication interface 1003 is configured to support the computing device 1000 to communicate, for example, to receive or send data or signals, etc.

[0270] The processor 1001 can be a central processing unit, a graphics processing unit (GPU), a general purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. The processor can also be a combination of components implementing computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc. The processor 1001 can be configured to read a program stored in the above-mentioned memory 1002 and execute the program. Figure 2 The operations performed by the computing device in the method described in the possible embodiments.

[0271] The embodiments of the present application also provide a device, which comprises a processor and a communication interface, and the device is configured to perform the method described in the above Figure 2 and the possible embodiments thereof.

[0272] In one possible embodiment, the device is a chip or a system on a chip (SoC). The embodiments of the present application also provide a device, which comprises a processor and a communication interface, and the device is configured to perform the method described in the above Figure 5 or Figure 6 and the possible embodiments thereof.

[0273] In one possible embodiment, the device is a chip or a system on a chip (SoC).

[0274] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method described in the above Figure 2 and the possible embodiments thereof.

[0275] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method described in the above Figure 5 or Figure 6 and the possible embodiments thereof.

[0276] The embodiments of the present application also provide a computer program product, and when the computer program product is read and executed by a computer, the method described in the above Figure 2 and the possible embodiments thereof will be executed.

[0277] The embodiments of the present application also provide a computer program product, and when the computer program product is read and executed by a computer, the method described in the above Figure 5 orFigure 6 The method described in the embodiments of the application and possible embodiments thereof will be executed.

[0278] The embodiments of the application also provide a computer program which, when executed on a computer, will cause the computer to implement the method described above Figure 2 The method described in the embodiments of the application and possible embodiments thereof will be executed.

[0279] The embodiments of the application also provide a computer program which, when executed on a computer, will cause the computer to implement the method described above Figure 5 or Figure 6 The method described in the embodiments of the application and possible embodiments thereof will be executed.

[0280] In summary, the depth nonlinear error is a nonlinear error caused by the fact that the waveform of the light signal is not a standard waveform (for example, is not a standard sine wave, etc.), and the error amount of the nonlinear error oscillates with distance changes, with positive and negative error amounts. Therefore, in the application, by sampling multiple times in the case of delaying the start of the light source than the sensor of the shooting device and performing the mean processing according to the n initial depth images, the positive and negative error amounts in the nonlinear error can be offset, so that the depth nonlinear error introduced in the measurement process can be reduced. At the same time, the field phase correction coefficient matrix obtained based on the sampling and mean processing operations can correct the depth value of each pixel in the depth image, reduce the error caused by the field phase, and greatly reduce the distortion rate of the final obtained depth image, thereby improving the quality of the depth image.

[0281] The terms "first", "second", and the like are used herein to distinguish between elements having substantially the same functions and / or actions and should be understood that there is no logical or chronological dependency between the "first", "second", "n-th" and the like. The quantity and the execution order of the elements should not be limited. It should also be understood that although the following description uses the terms first, second, and the like to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of various described examples, a first image can be referred to as a second image, and similarly, a second image can be referred to as a first image. The first image and the second image can both be images, and in some cases, can be separate and distinct images.

[0282] It should also be understood that in various embodiments of the application, the size of the serial number of various processes does not mean the order of execution, and the execution order of the processes should be determined by their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the application.

[0283] It should also be understood that the terms "comprises", "comprising", "includes", "including", "comprise", "comprising", "comprised of" and / or "comprising" when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0284] It should also be understood that the reference herein to "one embodiment", "an embodiment", "one possible implementation", "possible implementation" means that a particular feature, structure, or characteristic described in connection with an embodiment or implementation is included in at least one embodiment of the application. The appearances of the phrases "in one embodiment" or "in an embodiment", "in one possible implementation" in various places in the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0285] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, rather than limiting the technical solutions of the present application; even though the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the technical solutions recorded in the above-mentioned embodiments can be modified, or some or all of the technical features can be replaced equivalently; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of depth image processing, characterized by, The method comprises: acquiring a first depth image of a first object; correcting a depth value of a pixel in the first depth image by a field phase correction coefficient matrix to obtain a second depth image, wherein the field phase correction coefficient matrix is a matrix for correcting a field phase error obtained by preprocessing n initial depth images, the n initial depth images are depth images obtained in a first condition, the first condition is that a starting time of a light source of a shooting device and a starting time of a sensor of the shooting device are different time points, the preprocessing comprises mean processing performed on the n initial depth images, and n is an integer greater than 1; the n initial depth images are depth images of a second object obtained by n times of time-of-flight (TOF) ranging based on the shooting device, a starting time of the light source in the i th time of the n times is delayed by (i-1)*Δt from a starting time of the sensor, i is in a range of [1, n], n*|Δt|=k*T, T is a period of a light signal emitted by the light source, k is an integer, and Δt is a preset time length.

2. The method of claim 1, wherein, the preprocessing further comprises curved surface fitting processing, the field phase correction coefficient matrix is calculated according to n fitted depth images, and the n fitted depth images are respectively obtained by curved surface fitting on the n initial depth images.

3. The method of claim 2, wherein, the initial depth image obtained in the i th time is an i th initial depth image, the i th initial depth image is represented by a pixel matrix Di(x, y), each pixel value in a pixel matrix Di(x, y)' is summed with c*[(1-i)*Δt] / 2 to obtain the pixel matrix Di(x, y), the Di(x, y)' is a pixel matrix of a depth image obtained in the i th measurement, and c*[(1-i)*Δt] / 2 is a distance difference caused by the fact that the starting time of the light source is delayed by (i-1)*Δt from the starting time of the sensor.

4. The method according to claim 2 or 3, characterized in that, the field phase correction coefficient matrix is calculated according to n fitted depth images, and the n fitted depth images are respectively obtained by curved surface fitting on the n initial depth images. the field phase correction coefficient matrix is a ratio matrix obtained by respectively taking a ratio of a minimum value in a plurality of pixel values of a fitted average pixel matrix and each pixel value of the fitted average pixel matrix, and the fitted average pixel matrix is obtained by averaging pixel matrices of the n fitted depth images.

5. The method according to any of claims 2 or 3, characterized in that, The method further comprises: correcting the second depth image by a fixed pattern noise (FPN) matrix to obtain a third depth image, wherein each value in the FPN matrix comprises a fixed noise of a pixel in the first depth image with the same subscript as the each value caused by hardware.

6. The method of claim 5, wherein, The FPN matrix is a difference value matrix obtained by taking a difference value of a fitted average pixel matrix and an initial average pixel matrix, the fitted average pixel matrix is obtained by averaging pixel matrices of the n fitted depth images, and the initial average pixel matrix is obtained by averaging pixel matrices of the n initial depth images.

7. The method according to any one of claims 1 to 3, characterized in that, the correction of the first depth image by the field phase correction coefficient matrix to obtain the second depth image comprises: The product of the pixel matrix of the first depth image and the pixel value with the same subscript in the field phase correction coefficient matrix is calculated to obtain the second depth image.

8. The method of claim 5, wherein, The third depth image is obtained by correcting the second depth image through the fixed pattern noise (FPN) matrix, and the method comprises the following steps: The product of the pixel matrix of the first depth image and the pixel value with the same subscript in the field phase correction coefficient matrix is calculated to obtain a product matrix. The sum of the product matrix and the FPN matrix is calculated to obtain the third depth image.

9. A method of depth image processing, characterized by, The method comprises: n initial depth images are acquired, the n initial depth images being depth images of a second object acquired by a shooting device through n times of time-of-flight (TOF) ranging, wherein the starting time of a light source of the shooting device in the i-th time is delayed from the starting time of a sensor of the shooting device by (i-1)*Δt, i is an integer in the range of [1, n], n*|Δt|=k*T, T is the period of a light signal emitted by the light source, n is an integer greater than 1, k is an integer, and Δt is a preset time length; n fitted depth images are obtained by fitting the n initial depth images respectively; a field phase correction coefficient matrix is calculated according to the n fitted depth images after mean value processing, and the field phase correction coefficient matrix is used to correct the field phase error of a depth image acquired by the shooting device through the TOF ranging.

10. The method of claim 9, wherein, The initial depth image obtained in the i-th time is the i-th initial depth image, and the n initial depth images are acquired by: n measurement depth images actually measured by the shooting device through n times of time-of-flight (TOF) ranging are acquired, and the i-th measurement depth image is represented by a pixel matrix Di(x, y)'; the sum of each pixel value in the pixel matrix Di(x, y)' and c*[(1-i)*Δt] / 2 is calculated to obtain the i-th initial depth image Di(x, y), wherein c*[(1-i)*Δt] / 2 is the distance difference caused by the fact that the starting time of the light source is delayed from the starting time of the sensor by (i-1)*Δt.

11. The method according to claim 9 or 10, characterized in that, The field phase correction coefficient matrix is calculated according to the n fitted depth images after mean value processing, and the method comprises the following steps: a fitted average pixel matrix is obtained by averaging the pixel matrices of the n fitted depth images; the minimum value of a plurality of pixel values in the fitted average pixel matrix is extracted, and the ratio of the minimum value to each pixel value in the fitted average pixel matrix is calculated to obtain a ratio matrix, wherein the ratio matrix is the field phase correction coefficient matrix.

12. The method according to any of claims 9 or 10, characterized in that, After the n fitted depth images are obtained by fitting the n initial depth images respectively, the method further comprises the following steps: an initial average pixel matrix is obtained by averaging the pixel matrices of the n initial depth images; a fitted average pixel matrix is obtained by averaging the pixel matrices of the n fitted depth images; A difference matrix is obtained by taking a difference between the fitted average pixel matrix and the initial average pixel matrix, the difference matrix being a fixed pattern noise (FPN) matrix, each value in the FPN matrix including a fixed noise of a pixel in the depth image caused by hardware at a same subscript as the value.

13. A depth image processing device, characterized by, The device comprises: An acquisition unit configured to acquire a first depth image of a first object; A correction unit configured to correct a depth value of a pixel in the first depth image by a field phase correction coefficient matrix to obtain a second depth image, wherein the field phase correction coefficient matrix is a matrix for correcting a field phase error obtained by preprocessing n initial depth images, the n initial depth images being depth images of a second object obtained by a time-of-flight (TOF) ranging method through the imaging device n times, a start time of a light source in the i-th time being delayed from a start time of a sensor by (i-1)*Δt, i being an integer in a range of [1, n], n*|Δt|=k*T, T being a period of a light signal emitted by the light source, k being an integer, and Δt being a preset time length. The preprocessing further comprises surface fitting processing, 14. The apparatus of claim 13, wherein, The field phase correction coefficient matrix is obtained according to n fitted depth images, the n fitted depth images being obtained by surface fitting the n initial depth images, respectively. The i-th obtained initial depth image is an i-th initial depth image, the i-th initial depth image being represented by a pixel matrix Di(x, y), each pixel value in the pixel matrix Di(x, y) being a sum of a pixel value in a pixel matrix Di(x, y)' and c*[(1-i)*Δt] / 2, the Di(x, y)' being a pixel matrix of a depth image obtained by the i-th measurement, the c*[(1-i)*Δt] / 2 being a distance difference caused by the start time of the light source being delayed from the start time of the sensor by the (i-1)*Δt.

15. The apparatus of claim 14, wherein, The field phase correction coefficient matrix is obtained according to n fitted depth images, comprising:

16. The apparatus of claim 14 or 15, wherein, The field phase correction coefficient matrix is a ratio matrix obtained by taking a ratio of a minimum value of a plurality of pixel values in a fitted average pixel matrix and each pixel value in the fitted average pixel matrix, the fitted average pixel matrix being obtained by averaging pixel matrices of the n fitted depth images. The correction unit is further configured to:

17. The apparatus of any one of claims 14 or 15, wherein, correct the second depth image by a fixed pattern noise (FPN) matrix to obtain a third depth image, each value in the FPN matrix including a fixed noise of a pixel in the first depth image caused by hardware at a same subscript as the value. ​ 18. The apparatus of claim 17, wherein, The FPN matrix is a difference matrix obtained by taking a difference between a fitting average pixel matrix and an initial average pixel matrix, the fitting average pixel matrix is obtained by averaging pixel matrices of the n fitting depth images, and the initial average pixel matrix is obtained by averaging pixel matrices of the n initial depth images.

19. The apparatus of any one of claims 13 to 15, wherein, The correction unit is specifically used for: The second depth image is obtained by respectively calculating products of pixel matrices of the first depth image and pixel values of the same subscript in the field phase correction coefficient matrix.

20. The apparatus of claim 17, wherein, The correction unit is specifically used for: The product matrix is obtained by respectively calculating products of pixel matrices of the first depth image and pixel values of the same subscript in the field phase correction coefficient matrix. The third depth image is obtained by calculating a sum of the product matrix and the FPN matrix.

21. A depth image processing device, characterized by, The device comprises: An acquisition unit is configured to acquire n initial depth images of a second object, the n initial depth images being obtained by a shooting device through a time-of-flight (TOF) ranging method n times, wherein a starting time of a light source of the shooting device in the ith time is delayed from a starting time of a sensor of the shooting device by (i-1)*Δt, i is an integer in a range of [1, n], n*|Δt|=k*T, T is a period of a light signal emitted by the light source, n is an integer greater than 1, k is an integer, and Δt is a preset time length; A curved surface fitting unit is configured to perform curved surface fitting on the n initial depth images to obtain n fitting depth images; A calculation unit is configured to calculate a field phase correction coefficient matrix according to the n fitting depth images after mean value processing, the field phase correction coefficient matrix being used to correct a field phase error of a depth image obtained by the shooting device through the TOF ranging method.

22. The apparatus of claim 21, wherein, The ith initial depth image is the ith initial depth image; and the acquisition unit is specifically configured to: Acquire n measurement depth images actually measured by the shooting device through the TOF ranging method n times, an ith measurement depth image in the n measurement depth images being represented by a pixel matrix Di(x, y)'; Calculate a sum of each pixel value in the pixel matrix Di(x, y)' and c*[(1-i)*Δt] / 2 to obtain the ith initial depth image Di(x, y), c*[(1-i)*Δt] / 2 being a distance difference caused by the fact that the starting time of the light source is delayed from the starting time of the sensor by the (i-1)*Δt.

23. The apparatus of claim 21 or 22, wherein, The calculation unit is specifically configured to: Average pixel matrices of the n fitting depth images to obtain a fitting average pixel matrix; Extract a minimum value of a plurality of pixel values in the fitting average pixel matrix, and calculate ratios of the minimum value to respective pixel values of the fitting average pixel matrix to obtain a ratio matrix, the ratio matrix being the field phase correction coefficient matrix.

24. The apparatus of any one of claims 21 or 22, wherein, The computing unit is further configured to average pixel matrices of the n initial depth images to obtain an initial average pixel matrix after the surface fitting unit performs surface fitting on the n initial depth images respectively to obtain n fitted depth images; average pixel matrices of the n fitted depth images to obtain a fitted average pixel matrix; calculate a difference value between the fitted average pixel matrix and the initial average pixel matrix to obtain a difference value matrix, the difference value matrix being a fixed pattern noise (FPN) matrix, each value in the FPN matrix including a fixed noise of a pixel in a depth image with a same subscript as the value caused by hardware.

25. A depth image processing device, characterized by, The device comprises a processor, a communication interface and a memory, wherein the memory is configured to store a computer program, and the processor is configured to execute the computer program stored in the memory to implement the method according to any one of claims 1 to 8; or the processor is configured to execute the computer program stored in the memory to implement the method according to any one of claims 9 to 12.

26. A chip comprising a processor and a communications interface, characterized in that, The chip is configured to execute the method according to any one of claims 1 to 8 or any one of claims 9 to 12.

27. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 8 or any one of claims 9 to 12.

28. A computer program product, characterised in that, When the computer program product is read and executed by a computer, the method according to any one of claims 1 to 8 or any one of claims 9 to 12 will be executed.

Images