Depth image processing method and device and electronic equipment

Abstract

The invention discloses a depth image processing method and device and electronic equipment, and the method comprises the steps: obtaining a phase image set of a to-be-processed image and an original infrared image, and the phase image set at least comprises two phase images; obtaining a first depth image of the to-be-processed image based on phase data of the phase image set; performing curve processing on the original infrared image to obtain a first infrared image; performing pixel alignment processing on the first depth image and the first infrared image to obtain a second depth image and a second infrared image; gradient information of the second infrared image is obtained through calculation; and processing the second depth image based on the gradient information to obtain a target depth image corresponding to the to-be-processed image. According to the method, the gradient information of the infrared image is utilized to optimize the depth image, the problem of object edge detail loss in the depth image is solved, and the precision of the depth image is improved.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and particularly to a depth image processing method, apparatus, and electronic device. Background Art

[0002] With the development of three-dimensional imaging technology, Time of Flight (ToF) cameras are also applied to more and more scenarios, such as face recognition, gesture recognition, and pose recognition. A ToF camera calculates the depth information of a measured object by measuring the time for a laser to travel from a transmitter to the measured object and then to a receiver. However, during this measurement process, the ToF camera is also affected by many factors. For example, if the measured object moves relatively fast, the accuracy of the calculated depth information will be low. At this time, the motion image of the measured object will be relatively blurred or have motion artifacts, which cannot meet the requirements of the application scenario. Summary of the Invention

[0003] In view of the above problems, the present application provides a depth image processing method, apparatus, and electronic device, which improve the accuracy of depth images obtained by a ToF camera.

[0004] To achieve the above object, the present application provides the following technical solutions:

[0005] A depth image processing method, the method comprising:

[0006] Obtaining a set of phase images and an original infrared image of an image to be processed, the set of phase images including at least two phase images;

[0007] Obtaining a first depth image of the image to be processed based on the phase data of the set of phase images;

[0008] Performing curve processing on the original infrared image to obtain a first infrared image;

[0009] Performing pixel alignment processing on the first depth image and the first infrared image to obtain a second depth image and a second infrared image;

[0010] Calculating gradient information of the second infrared image;

[0011] Processing the second depth image based on the gradient information to obtain a target depth image corresponding to the image to be processed.

[0012] Optionally, the obtaining a first depth image of the image to be processed based on the phase data of the set of phase images includes:

[0013] Calculate the signal intensity of each pixel point of the image to be processed based on the phase data of each phase image in the set of phase images;

[0014] Screen the signal intensities according to a signal intensity threshold to obtain the screened signal intensities;

[0015] Calculate the depth information of the pixel points corresponding to the screened signal intensities based on the screened signal intensities;

[0016] Obtain a first depth image of the image to be processed based on the depth information of each pixel point.

[0017] Optionally, the processing the original infrared image by a curve to obtain a first infrared image includes:

[0018] Perform an S-curve processing on each pixel point in the original infrared image to obtain the processed pixel points;

[0019] Generate a first infrared image based on each of the processed pixel points.

[0020] Optionally, the processing of pixel alignment of the first depth image and the first infrared image to obtain a second depth image and a second infrared image includes:

[0021] Based on a first intrinsic matrix corresponding to the first depth image, correct the positions of each pixel point in the first depth image to obtain a first set of pixel points corresponding to the first depth image;

[0022] Based on a second intrinsic matrix corresponding to the first infrared image, correct the positions of each pixel point in the first infrared image to obtain a second set of pixel points corresponding to the first infrared image;

[0023] Based on a first extrinsic parameter corresponding to the first depth image and a second extrinsic parameter corresponding to the first infrared image, map each pixel point of the first set of pixel points and the second set of pixel points to the same space to obtain a second depth image and a second infrared image.

[0024] Optionally, the calculating the gradient information of the second infrared image includes:

[0025] For each pixel group of the second infrared image, calculate a first derivative in the horizontal direction and a second derivative in the vertical direction;

[0026] Based on the first derivative and the second derivative, determine the gradient information of the second infrared image, where the gradient information includes a gradient value and a gradient direction.

[0027] Optionally, the processing of the second depth image based on the gradient information to obtain a target depth image corresponding to the image to be processed includes:

[0028] Processing the second depth image based on the gradient information to obtain a third depth image;

[0029] Processing the non-edge region of the third depth image to obtain a target depth image corresponding to the image to be processed.

[0030] Optionally, the processing of the second depth image based on the gradient information to obtain a third depth image includes:

[0031] Performing local linear transformation on the second depth image based on the gradient information to obtain a third infrared image;

[0032] Adjusting the pixel data in the third infrared image based on a target loss function so that the similarity between the pixel data in the adjusted third infrared image and the pixel data in the second depth image meets a similarity threshold;

[0033] Determining a third depth image based on the adjusted third infrared image.

[0034] Optionally, the processing of the non-edge region of the third depth image to obtain a target depth image corresponding to the image to be processed includes:

[0035] Obtaining the non-edge region of the third depth image;

[0036] Restoring the depth data of the image of the non-edge region of the third depth image based on the depth information of the non-edge region of the second depth image to obtain an image of the target non-edge region;

[0037] Generating a target depth image corresponding to the image to be processed based on the edge region image of the third depth image and the image of the target non-edge region.

[0038] A depth image processing apparatus, the method apparatus includes:

[0039] A first acquisition unit, configured to acquire a phase image set and an original infrared image of an image to be processed, where the phase image set includes at least two phase images;

[0040] A second acquisition unit, configured to acquire a first depth image of the image to be processed based on the phase data of the phase image set;

[0041] A first processing unit, configured to perform curve processing on the original infrared image to obtain a first infrared image;

[0042] A second processing unit, configured to perform pixel alignment processing on the first depth image and the first infrared image to obtain a second depth image and a second infrared image;

[0043] A calculation unit, configured to calculate and obtain gradient information of the second infrared image;

[0044] A third processing unit, configured to process the second depth image based on the gradient information to obtain a target depth image corresponding to the image to be processed.

[0045] An electronic device, comprising:

[0046] A memory, configured to store a program;

[0047] A processor, configured to execute the program, and the program is specifically configured to implement the depth image processing method described in any one of the above.

[0048] Compared with the prior art, the present application provides a depth image processing method, apparatus and electronic device. The method includes: obtaining a phase image set and an original infrared image of an image to be processed, where the phase image set includes at least two phase images; obtaining a first depth image of the image to be processed based on the phase data of the phase image set; performing curve processing on the original infrared image to obtain a first infrared image; performing pixel alignment processing on the first depth image and the first infrared image to obtain a second depth image and a second infrared image; calculating and obtaining gradient information of the second infrared image; processing the second depth image based on the gradient information to obtain a target depth image corresponding to the image to be processed. The present application optimizes the depth map using the gradient information of the infrared image, solves the loss of object edge details in the depth map, and improves the accuracy of the depth image. Description of the Drawings

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0050] Figure 1 It is a schematic flowchart of a depth image processing method provided by an embodiment of the present application;

[0051] Figure 2 It is a schematic diagram of the order of obtaining the phase image and the infrared image of the image to be processed provided by an embodiment of the present application;

[0052] Figure 3Schematic diagram of an S curve for optimizing an original infrared image provided by an embodiment of the present application;

[0053] Figure 4 Comparison schematic diagram before and after depth image optimization processing provided by an embodiment of the present application;

[0054] Figure 5 System schematic diagram of a ToF camera provided by an embodiment of the present application;

[0055] Figure 6 Structural schematic diagram of a depth image processing device provided by an embodiment of the present application. Specific implementation manners

[0056] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0057] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned accompanying drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may include unlisted steps or units.

[0058] In an embodiment of the present application, a depth image processing method is provided to improve the accuracy of a depth image obtained by a ToF camera. Refer to Figure 1 , which is a flow schematic diagram of a depth image processing method provided by an embodiment of the present application. The method may include the following steps:

[0059] S101. Obtain a phase image set and an original infrared image of the image to be processed.

[0060] The image to be processed refers to an initial depth image, which can be collected by a time-of-flight (ToF) camera. The ToF camera calculates the depth information of the object to be measured by measuring the time it takes for the laser to travel from the transmitter to the object to be measured and then to the receiver. The object to be measured refers to the object for which a depth image needs to be captured, which can be a stationary object or a moving object. In an embodiment of the present application, the object to be measured mainly refers to a moving object. Furthermore, the depth image processing method provided by the embodiment of the present application can solve the problem of void regions in the depth image due to low accuracy when the object to be measured is a moving object.

[0061] A set of phase images corresponding to the image to be processed can be obtained through a time-of-flight (ToF) camera. The set of phase images includes at least two phase images. For example, the ToF camera can be set with a modulation frequency and a data acquisition order of acquiring four frames of phase maps, and then a single-frame original infrared image (represented as an IR image) is obtained through an infrared camera. It should be noted that in the embodiments of the present application, the number of phase images in the set of phase images is not limited. However, if the number is small, it will affect the accuracy of subsequent calculations. If the number is too large, higher requirements will be imposed on the sensor, and the calculation complexity will be greater, occupying too much computing resources. Therefore, four frames of phase images are preferably used in the present application.

[0062] For example, referring to Figure 2 , which is a schematic diagram of the order of obtaining the phase image and the infrared image of the image to be processed provided by the embodiments of the present application. Acquisition is performed by delaying 0°, 90°, 180°, and 270° respectively according to the initial phase, that is, measuring once every quarter cycle. The four frames of phase images of the image to be processed (i.e., the initial depth image) are respectively phase1, phase2, phase3, and phase4, and then a single-frame original infrared image (represented as an IR image) is obtained. The obtained set of phase images can be used to determine the signal intensity, facilitating the preliminary processing of the image to be processed and improving the accuracy of the image.

[0063] S102. Obtain a first depth image of the image to be processed based on the phase data of the set of phase images.

[0064] Through the phase data corresponding to the phase image, the depth information of the image to be processed can be screened, and the depth information of the pixel points with higher confidence can be obtained, thereby obtaining the first depth image. In an implementation manner of the embodiments of the present application, the process of obtaining the first depth image of the image to be processed based on the phase data of the set of phase images may include:

[0065] Calculate the signal intensity of each pixel point of the image to be processed according to the phase data of each phase image in the set of phase images; screen each signal intensity according to a signal intensity threshold to obtain the screened signal intensity; calculate the depth information of the pixel points corresponding to the screened signal intensity based on the screened signal intensity; and obtain the first depth information of the image to be processed based on the depth information of each pixel point.

[0066] Still taking Figure 2 the obtained four frames of phase images and one frame of infrared image as an example, the phase data corresponding to each frame of phase image is respectively represented as:

[0067]

[0068]

[0069] in, is the phase difference, a and b are constants, and A and B are the two taps of a single pixel of the current assisted photonic demodulator CAPD (Current Assisted Photonic Demodulators).

[0070] Calculate the cosine component I and sine component Q of each pixel and the signal strength amp according to the four frames of phase data:

[0071] I=phase1-phase3 (5)

[0072] Q=phase4-phase2 (6)

[0073]

[0074] The distance is calculated based on the four-frame phase data, and the phase difference is calculated based on I and Q:

[0075]

[0076] Then perform distance calculation to get the depth value of each pixel:

[0077]

[0078] Where c is the speed of light and Mod is the modulation frequency.

[0079] The depth value of each pixel can be calculated first, and then the depth values ​​can be filtered based on the signal strength. Alternatively, the signal strength can be filtered first, and then the depth value can be calculated based on the I and Q corresponding to the filtered signal strength. The latter can save some unnecessary pixel depth value calculation processes and save computing resources.

[0080] The signal strength threshold can be determined according to the actual application scenario. For example, the closer the object to be measured is to the camera, the stronger the signal strength is, and the farther the object to be measured is from the camera, the weaker the signal strength is. The weaker the signal, the greater the error is likely to be, and these weaker signals can be screened out. Therefore, the signal strength threshold can be determined according to the distance between the object to be measured and the camera. The depth information of the pixel points is screened according to the signal strength threshold s, and the depth information with higher confidence is retained, and the pixels with lower confidence are removed, so that the first depth image can be obtained according to the depth information of the remaining pixels, represented by D'. In addition, when the object to be measured moves faster, the confidence of the depth data at the edge of the object to be measured is low, and a hole area is easily formed, that is, the signal strength of the pixel points in the hole area is less than the intensity threshold, as follows:

[0081]

[0082] S103. Perform curve processing on the original infrared image to obtain the first infrared image.

[0083] In order to more prominently display the image information in the infrared image, it is necessary to perform curve processing on the original infrared image to obtain the first infrared image. For example, S-curve processing can be used, or other forms of curve processing can be selected in combination with the actual application scenario to highlight the gradient magnitude of the edge region of the infrared image.

[0084] S104. Perform pixel alignment processing on the first depth image and the first infrared image to obtain the second depth image and the second infrared image.

[0085] If the alignment of the depth image and the infrared image is not precise enough, it will cause a large error when the infrared image is subsequently used to optimize the depth image. Specifically, the pixel points of the depth image and the infrared image can be aligned by collecting the parameters of the acquisition modules of the depth image and the infrared image, so that each pixel point can be aligned based on the same spatial coordinate and acquisition dimension.

[0086] S105. Calculate the gradient information of the second infrared image.

[0087] The second infrared image is the infrared image aligned with the first depth image. The gradient information can provide information about the position of the edge region of the image and the different intensity information of the pixels on either side of the edge region. For example, the pixels in the edge region can be located at the local maximum of the gradient magnitude. Therefore, the gradient information of the second infrared image can be used to optimize the edge region of the depth image.

[0088] S106. Process the second depth image based on the gradient information to obtain the target depth image corresponding to the image to be processed.

[0089] The second depth image is the depth image aligned with the first infrared image. The gradient information of the second infrared image can be used to highlight the edge gradient of the second depth image, making the edge of the second depth image more obvious, compensating for the loss of details of the object edge in the depth map, providing more accurate depth data, and thus obtaining the target depth image.

[0090] An embodiment of the present application provides a depth image processing method, which includes obtaining a set of phase images and an original infrared image of an image to be processed, where the set of phase images includes at least two phase images; obtaining a first depth image of the image to be processed based on the phase data of the set of phase images; performing curve processing on the original infrared image to obtain a first infrared image; performing pixel alignment processing on the first depth image and the first infrared image to obtain a second depth image and a second infrared image; calculating the gradient information of the second infrared image; and processing the second depth image based on the gradient information to obtain a target depth image corresponding to the image to be processed. The present application optimizes the depth map using the gradient information of the infrared image, solves the loss of object edge details in the depth map, and improves the accuracy of the depth image.

[0091] The following describes the specific implementation manners of the relevant technical features of the embodiments of the present application in combination with actual application scenarios.

[0092] In order to highlight the gradient magnitude of the edge region of the infrared image, in an embodiment of the present application, curve processing may be performed on the original infrared image to obtain a first infrared image, and this process includes:

[0093] Performing S-curve processing on each pixel point in the original infrared image to obtain a processed pixel point; generating a first infrared image based on each processed pixel point.

[0094] The original infrared image obtained for a single frame (represented by Figure I) undergoes an S-curve transformation to process the global contrast of the original infrared image. The specific S-curve formula is as follows:

[0095]

[0096] where x is the pixel value input to the original infrared image, and y is the pixel value of the first infrared image after the S-curve transformation. When a takes 0.5 and γ takes 0.5, an S-curve graph can be obtained as Figure 3 shown, and the processed first infrared image is obtained, represented by Figure I'. Figure I' has an improved contrast compared to Figure I, which is beneficial for highlighting the gradient magnitude of the edge region.

[0097] In order to improve the optimization accuracy of the depth image, in an embodiment of the present application, the depth image and the infrared image also need to be aligned. Specifically, pixel alignment processing is performed on the first depth image and the first infrared image to obtain a second depth image and a second infrared image, including:

[0098] Based on the first intrinsic matrix corresponding to the first depth image, correct the positions of each pixel point in the first depth image to obtain a first pixel point set corresponding to the first depth image; based on the second intrinsic matrix corresponding to the first infrared image, correct the positions of each pixel point in the first infrared image to obtain a second pixel point set corresponding to the first infrared image; based on the first extrinsic parameter corresponding to the first depth image and the second extrinsic parameter corresponding to the first infrared image, map each pixel point in the first pixel point set and the second pixel point set to the same space to obtain a second depth image and a second infrared image.

[0099] First, perform intrinsic calibration. By using a checkerboard image, the first intrinsic matrix of the ToF camera that acquires the depth image and the second intrinsic matrix of the IR camera that acquires the infrared image can be calculated. The first intrinsic matrix is denoted as K T and the second intrinsic matrix is denoted as K I , and each of them includes the focal lengths (f x , f y ) and the principal points (c x , c y ).

[0100] Among them, the first intrinsic matrix K T of ToF can be expressed as:

[0101]

[0102] The second intrinsic matrix K I of the IR camera can be expressed as:

[0103]

[0104] Through intrinsic calibration, it is possible to correct the positions of pixel points in the image and avoid the deviation of pixel point positions.

[0105] Taking any pixel point in the first depth image as an example, if the coordinate of a pixel point in the first depth image coordinate system is P T =(x T , y T , z T ), normalizing it to the depth plane can obtain

[0106] Then, Using the first memory matrix K T of ToF to convert P T to the image coordinate system of the depth image, we get:

[0107]

[0108] In the above formula (14), (uT , v T ) is the corresponding coordinate point in the image coordinate system of the depth map corresponding to P T = (x T , y T , z T ). The depth image composed of each pixel point after the above conversion is used as the second depth image.

[0109] Similarly, taking any pixel point in the first infrared image as an example, assume that there is a point with coordinates p1 = (x1, y1, z1) in the coordinate system of the first infrared image. Normalizing it to the IR image plane can obtain

[0110] Then, using the second internal parameter matrix K of the IR camera for p1 = (x1, y1, z1) I Convert P I to the image coordinate system of the IR image:

[0111]

[0112] In the above formula (15), where (u1, v1) is the corresponding coordinate point in the IR coordinate system of the IR image corresponding to p1 = (x1, y1, z1). The infrared image corresponding to each pixel point after coordinate conversion is determined as the second infrared image.

[0113] When performing external parameter calibration, set the coordinate system of the first depth image as O T -X T Y T Z T , and the coordinate system of the IR image as O I -X I Y I Z I . Select a calibration board as an intermediate medium and set its reference coordinate system as O B -X B Y B Z B .

[0114] Calibrate the external parameters between the calibration board and the IR camera, including the rotation matrix R B2I and the translation matrix t B2I .

[0115]

[0116] Calibrate the external parameters between the calibration board and the ToF, including the rotation matrix R B2T and the translation matrix t B2T .

[0117]

[0118] The derivation of the ToF depth map to the IR map can be obtained according to Equations (16) and (17):

[0119]

[0120] where is the rotation matrix R from the depth map coordinate system O T -X T Y T Z T to the IR map coordinate system O I -X I Y I Z I , and T2I , is the translation matrix t from the ToF to the IR camera T2I .

[0121] Converting the pixel point (u T , v T ) in the depth map to the depth map coordinate system O T -X T Y T Z T , we get (x T , y T , z T ):

[0122]

[0123] Furthermore, using the rotation matrix R T2I and the translation matrix t T2I , the point (x T , y T , z T ) in the depth map coordinate system is converted to the IR map coordinate system, and we get (x I , y I , z I ):

[0124]

[0125] Projecting the obtained point (x I , y I , z I ) in the IR map coordinate system onto the IR image plane, we get the position (u I , v I ) of the pixel point in the IR image:

[0126]

[0127] The above Equation (21) is equivalent to using the intrinsic parameter matrix K I, the obtained formula (15):

[0128]

[0129] Through the adjustment based on the external parameters, the pixel points in the first infrared image and the first depth image can be mapped to the same spatial coordinates, realizing the alignment of the first infrared image and the first depth image, and obtaining the aligned second depth image and second infrared image.

[0130] In an implementation manner of the embodiment of the present application, calculating the gradient information of the second infrared image includes:

[0131] For each pixel group of the second infrared image, calculate the first derivative in the horizontal direction and the second derivative in the vertical direction; based on the first derivative and the second derivative, determine the gradient information of the second infrared image, and the gradient information includes the gradient value and the gradient direction.

[0132] For each pixel group of the second infrared image, derivative filters can be applied to calculate the first derivative in the horizontal and vertical directions respectively. The derivatives in the corresponding directions can represent the horizontal and vertical changes of the pixel intensity values, thereby generating a gradient vector value, including the gradient value and the gradient direction. Information about the position of the object boundary and the different intensity values of the regions of the pixels on either side of the boundary can be provided.

[0133] In an implementation manner, processing the second depth image based on the gradient information to obtain a target depth image corresponding to the image to be processed includes:

[0134] Processing the second depth image based on the gradient information to obtain a third depth image; processing the non-edge region of the third depth image to obtain a target depth image corresponding to the image to be processed.

[0135] Further, processing the second depth image based on the gradient information to obtain a third depth image includes:

[0136] Performing local linear transformation on the second depth image based on the gradient information to obtain a third infrared image;

[0137] Adjusting the pixel point data in the third infrared image based on the target loss function so that the similarity between the pixel point data in the adjusted third infrared image and the pixel point data in the second depth image meets the similarity threshold; determining the third depth image based on the adjusted third infrared image.

[0138] Further, processing the non-edge region of the third depth image to obtain a target depth image corresponding to the image to be processed includes:

[0139] Obtain the non-edge region of the third depth image; restore the depth data of the image of the non-edge region of the third depth image based on the depth information of the non-edge region of the second depth image to obtain the image of the target non-edge region; generate the target depth image corresponding to the image to be processed based on the edge region image of the third depth image and the image of the target non-edge region.

[0140] For example, process the aligned second infrared image (Figure I') to obtain its gradient information, and then process it according to the obtained aligned second depth image (Figure D'). It is necessary to find a third infrared image (Figure I") that is locally as consistent as possible with Figure I' in terms of gradient and similar to Figure D' in terms of data.

[0141] A sliding window w needs to be used when calculating the gradient of the second infrared image (Figure I'). k , and calculate the whole image successively. Here, a Gaussian filter kernel w is selected. ij (I′):

[0142]

[0143] where x and y are the horizontal distance and vertical distance of the current pixel in Figure I′ from the middle pixel respectively, σ is the standard deviation controlling the smoothness of the kernel, and k i is the normalization parameter, which can ensure that ∑ j w ij = 1.

[0144] Assume that the obtained image I″ is a local linear transformation of I′, and Figure I″ can be initially obtained:

[0145] I″ i = a k ·I′ i + b k , i ∈ w k (23)

[0146] where a k and b k are constant coefficients, which can make the filtered I″ be as consistent as possible with I′ in terms of gradient in local content.

[0147] To optimize the objective loss function so that the Figure I″ obtained from Equation (23) can be close to Figure D′ in terms of data, the objective loss function needs to be optimized as follows:

[0148]

[0149] where ε is a very small adjustment parameter, and the purpose is to prevent a kThere is a situation of division by zero. Therefore, the finally obtained Figure I″ can be similar to Figure I′ as a whole in terms of gradient, while being similar to Figure D′ in terms of data.

[0150] Furthermore, take the partial derivative of Equation (24):

[0151]

[0152] Let Equation (23) be zero, then:

[0153]

[0154] Let Equation (24) be zero, then:

[0155]

[0156] Substitute Equation (28) into Equation (26), then there is:

[0157]

[0158]

[0159] From the formulas of expectation, variance, and covariance, it can be obtained that:

[0160]

[0161] Therefore, α k and b k can be calculated.

[0162] Considering that the output of the filtering is a local linear transformation, for any point k, the points within its window w k can obtain a result of local linear transformation. When the sliding window w k moves, the same point k will go through multiple transformations to obtain different results. Therefore, the average value of multiple results can be taken to obtain the final result.

[0163]

[0164] The finally obtained processed I″ i and the required D″ i can be considered approximately equal:

[0165] I″ i ≈ D″ i (38)

[0166] That is, the third infrared image I” and the third depth image D” are obtained.

[0167] After obtaining the third depth image D″, the processing of non-edge regions is included. However, in practice, it should be considered that the depth data of non-edge regions is highly reliable and does not require optimization processing. Therefore, the depth data of non-edge regions is restored.

[0168]

[0169] After restoring the depth data of non-edge regions, the final target depth image can be obtained.

[0170] See Figure 4 , which is a comparison schematic diagram before and after depth image optimization processing provided by an embodiment of the present application. Figure 4 (a) is the depth gesture data before optimization. For easy viewing, it is converted to 8-bit data. Figure 4 (b) is the gesture data in the IR image after S-curve transformation. Figure 4 (c) is the optimized depth gesture data.

[0171] See Figure 5 , which is a system schematic diagram of a ToF camera provided by an embodiment of the present application, including a transmitting module, a receiving module, a control module, and a calculation module. The control module 61 controls the transmitting module 63 and the receiving module 64. The signals output by the control module 61 to the transmitting module 63 include the modulation period of the optical signal, and the signals output to the receiving module include the timing of triggering reception. The transmitting module is used to transmit an optical signal with a specified modulation period, and the receiving module is used to collect the optical signal within a specified time. The calculation module is used for operations such as distance calculation, infrared image gradient calculation, and depth image optimization. Specifically, the calculation module can be used to execute the depth image processing method provided in the above embodiment.

[0172] See Figure 6 , which is a structural schematic diagram of a depth image processing device provided by an embodiment of the present application. The device may include:

[0173] The first acquisition unit 601 is used to obtain a set of phase images and an original infrared image of the image to be processed, and the set of phase images includes at least two phase images;

[0174] The second acquisition unit 602 is used to obtain the first depth image of the image to be processed based on the phase data of the set of phase images;

[0175] The first processing unit 603 is used to perform curve processing on the original infrared image to obtain a first infrared image;

[0176] The second processing unit 604 is used to perform pixel alignment processing on the first depth image and the first infrared image to obtain a second depth image and a second infrared image;

[0177] A calculation unit 605 for calculating and obtaining gradient information of the second infrared image;

[0178] A third processing unit 606 for processing the second depth image based on the gradient information to obtain a target depth image corresponding to the image to be processed.

[0179] Optionally, the second acquisition unit includes:

[0180] A first calculation subunit for calculating and obtaining the signal intensity of each pixel point of the image to be processed according to the phase data of each phase image in the phase image set;

[0181] A first screening subunit for screening each of the signal intensities according to a signal intensity threshold to obtain screened signal intensities;

[0182] A second calculation subunit for calculating depth information of pixel points corresponding to the screened signal intensities based on the screened signal intensities;

[0183] A first acquisition subunit for obtaining a first depth image of the image to be processed based on the depth information of each pixel point.

[0184] Optionally, the first processing unit:

[0185] A first processing subunit for performing an S-curve processing on each pixel point in the original infrared image to obtain processed pixel points;

[0186] A generation subunit for generating a first infrared image based on each of the processed pixel points.

[0187] Optionally, the second processing unit includes:

[0188] A first correction subunit for correcting the positions of each pixel point in the first depth image based on a first internal parameter matrix corresponding to the first depth image to obtain a first pixel point set corresponding to the first depth image;

[0189] A second correction subunit for correcting the positions of each pixel point in the first infrared image based on a second internal parameter matrix corresponding to the first infrared image to obtain a second pixel point set corresponding to the first infrared image;

[0190] A mapping subunit for mapping each pixel point of the first pixel point set and the second pixel point set to the same space based on a first external parameter corresponding to the first depth image and a second external parameter corresponding to the first infrared image to obtain a second depth image and a second infrared image.

[0191] Optionally, the computing unit includes:

[0192] A third computing subunit, configured to calculate a first derivative in the horizontal direction and a second derivative in the vertical direction for each pixel group of the second infrared image;

[0193] A first determining subunit, configured to determine gradient information of the second infrared image based on the first derivative and the second derivative, where the gradient information includes a gradient value and a gradient direction.

[0194] Optionally, the third processing unit includes:

[0195] A second processing subunit, configured to process the second depth image based on the gradient information to obtain a third depth image;

[0196] A third processing subunit, configured to process a non-edge region of the third depth image to obtain a target depth image corresponding to the image to be processed.

[0197] Optionally, the second processing subunit is configured to:

[0198] Perform a local linear transformation on the second depth image based on the gradient information to obtain a third infrared image;

[0199] Adjust pixel point data in the third infrared image based on a target loss function, so that the similarity between the pixel point data in the adjusted third infrared image and the pixel point data in the second depth image meets a similarity threshold;

[0200] Determine a third depth image based on the adjusted third infrared image.

[0201] Optionally, the third processing subunit is configured to

[0202] Obtain a non-edge region of the third depth image;

[0203] Restore depth data of an image of the non-edge region of the third depth image based on depth information of the non-edge region of the second depth image to obtain an image of a target non-edge region;

[0204] Generate a target depth image corresponding to the image to be processed based on an edge region image of the third depth image and the image of the target non-edge region.

[0205] Based on the foregoing embodiments, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the depth image processing method as described in any one of the foregoing.

[0206] An embodiment of the present application further provides an electronic device, including:

[0207] A memory for storing a program;

[0208] A processor for executing the program, and the program is specifically used to implement the deep image processing method described in any one of the above.

[0209] It should be noted that the above processor or CPU can be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that the electronic device implementing the above processor function can also be others, and the embodiments of the present application do not make specific limitations.

[0210] It should be noted that the above computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it can also be various terminals including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0211] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method section.

[0212] The foregoing description of the disclosed embodiments enables those skilled in the art to practice or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A depth image processing method, characterized in that: The method comprises: Obtaining a phase image set and an original infrared image of an image to be processed, wherein the phase image set includes at least two phase images; Based on the phase data of the phase image set, obtaining a first depth image of the image to be processed; Performing curve processing on the original infrared image to obtain a first infrared image; Performing pixel alignment processing on the first depth image and the first infrared image to obtain a second depth image and a second infrared image; Calculating and obtaining gradient information of the second infrared image; The second depth image is processed based on the gradient information to obtain a target depth image corresponding to the image to be processed.

2. The depth image processing method according to claim 1, characterized in that: The step of obtaining a first depth image of the image to be processed based on the phase data of the phase image set comprises: Calculating the signal intensity of each pixel of the image to be processed according to the phase data of each phase image in the phase image set; Screening each of the signal strengths according to a signal strength threshold to obtain a screened signal strength; Based on the filtered signal strength, calculating the depth information of the pixel points corresponding to the filtered signal strength; Based on the depth information of each pixel, a first depth image of the image to be processed is obtained.

3. The depth image processing method according to claim 1, characterized in that: The step of performing curve processing on the original infrared image to obtain a first infrared image includes: Performing S-curve processing on each pixel in the original infrared image to obtain a processed pixel; Based on each of the processed pixel points, a first infrared image is generated.

4. The depth image processing method according to claim 1, characterized in that: The performing pixel alignment processing on the first depth image and the first infrared image to obtain a second depth image and a second infrared image includes: Based on a first intrinsic parameter matrix corresponding to the first depth image, correcting the position of each pixel point in the first depth image to obtain a first pixel point set corresponding to the first depth image; Based on a second internal parameter matrix corresponding to the first infrared image, correcting the position of each pixel point in the first infrared image to obtain a second pixel point set corresponding to the first infrared image; Based on a first extrinsic parameter corresponding to the first depth image and a second extrinsic parameter corresponding to the first infrared image, each pixel point of the first pixel point set and the second pixel point set are mapped to the same space to obtain a second depth image and a second infrared image.

5. The depth image processing method according to claim 1, characterized in that: The calculating and obtaining the gradient information of the second infrared image includes: For each pixel group of the second infrared image, calculating a first derivative in a horizontal direction and a second derivative in a vertical direction; Based on the first derivative and the second derivative, gradient information of the second infrared image is determined, where the gradient information includes a gradient value and a gradient direction.

6. The depth image processing method according to claim 1, characterized in that: The processing of the second depth image based on the gradient information to obtain a target depth image corresponding to the image to be processed includes: Processing the second depth image based on the gradient information to obtain a third depth image; The non-edge area of ​​the third depth image is processed to obtain a target depth image corresponding to the image to be processed.

7. The depth image processing method according to claim 6, characterized in that: The processing of the second depth image based on the gradient information to obtain a third depth image includes: Performing a local linear change on the second depth image based on the gradient information to obtain a third infrared image; Adjusting the pixel data in the third infrared image based on the target loss function so that the similarity between the pixel data in the adjusted third infrared image and the pixel data in the second depth image meets a similarity threshold; Based on the adjusted third infrared image, a third depth image is determined.

8. The depth image processing method according to claim 6, characterized in that: The processing of the non-edge area of ​​the third depth image to obtain a target depth image corresponding to the image to be processed includes: Obtaining a non-edge area of ​​the third depth image; Performing depth data restoration on an image of a non-edge area of ​​the third depth image based on the depth information of the non-edge area of ​​the second depth image to obtain an image of a target non-edge area; Based on the edge area image of the third depth image and the image of the target non-edge area, a target depth image corresponding to the image to be processed is generated.

9. A depth image processing device, characterized in that: The method device comprises: A first acquisition unit, used to obtain a phase image set and an original infrared image of an image to be processed, wherein the phase image set includes at least two phase images; A second acquisition unit, configured to obtain a first depth image of the image to be processed based on the phase data of the phase image set; A first processing unit, configured to perform curve processing on the original infrared image to obtain a first infrared image; A second processing unit, configured to perform pixel alignment processing on the first depth image and the first infrared image to obtain a second depth image and a second infrared image; A calculation unit, used for calculating and obtaining gradient information of the second infrared image; The third processing unit is used to process the second depth image based on the gradient information to obtain a target depth image corresponding to the image to be processed.

10. An electronic device, characterized in that: include: Memory, used to store programs; A processor is used to execute the program, wherein the program is specifically used to implement the depth image processing method as described in any one of claims 1 to 8.

Images