Depth information optimization method, device and electronic device

Abstract

The present application discloses a method, device and electronic device for optimizing depth information, which belongs to image processing technology. The method comprises: determining a first depth matrix, which is a mixed depth matrix of a time-of-flight TOF camera under a first frequency signal and a second frequency signal, wherein the frequency of the first frequency signal is higher than the frequency of the second frequency signal; obtaining a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera; optimizing the first depth matrix based on the first confidence matrix and the initial intrinsic parameter matrix.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a depth information optimization method, device and electronic device. Background Art

[0002] At present, depth perception technology has made great progress in mobile phones, autonomous driving, and scene reconstruction. There are two mainstream ways to obtain depth information: active depth measurement and passive depth measurement. Active depth measurement mainly includes 3D structured light and time-of-flight (TOF) ranging methods. Due to its compact structure, long working distance, wide application scenarios, and high accuracy at longer distances, TOF cameras are increasingly widely used. TOF cameras use the TOF ranging principle to determine the distance between the camera and the object or the surrounding environment, that is, the TOF sensor measures the time it takes for the emitted light to reach the object and reflect back to the TOF sensor, and converts it into distance, and finally generates a depth image or 3D image through the measured points.

[0003] However, in related technologies, the collected depth data may contain noise, or the depth data may be missing due to factors such as material, occlusion, and distance, forming holes, which makes the output depth map of low quality and has a great impact on subsequent recognition, classification and other tasks. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a depth information optimization method, device and electronic device, which can solve the problem of low quality of the output depth map.

[0005] In a first aspect, an embodiment of the present application provides a depth information optimization method, including: determining a first depth matrix, where the first depth matrix is ​​a mixed depth matrix of a time-of-flight TOF camera under a first frequency signal and a second frequency signal, and the frequency of the first frequency signal is higher than the frequency of the second frequency signal; obtaining a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera; and optimizing the first depth matrix based on the first confidence matrix and the initial intrinsic parameter matrix.

[0006] In a second aspect, an embodiment of the present application provides a depth information optimization device, including: a determination module, used to determine a first depth matrix, the first depth matrix being a mixed depth matrix of a time-of-flight TOF camera under a first frequency signal and a second frequency signal, the frequency of the first frequency signal being higher than the frequency of the second frequency signal; an acquisition module, used to acquire a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera; an optimization module, used to optimize the first depth matrix based on the first confidence matrix and the initial intrinsic parameter matrix.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0010] In an embodiment of the present application, by determining a first depth matrix, the first depth matrix is ​​a mixed depth matrix of a time-of-flight TOF camera under a first frequency signal and a second frequency signal, and the frequency of the first frequency signal is higher than the frequency of the second frequency signal; obtaining a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera; and optimizing the first depth matrix based on the first confidence matrix and the initial intrinsic parameter matrix, the precision and accuracy of the collected depth data can be improved, noise can be reduced, and the quality of the depth map can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a schematic flow chart of a depth information optimization method according to an embodiment of the present application;

[0012] Figure 2 is a schematic flow chart of a depth information optimization method according to another embodiment of the present application;

[0013] Figure 3 is a schematic flow chart of a depth information optimization method according to another embodiment of the present application;

[0014] Figure 4 is a schematic flow chart of a depth information optimization method according to another embodiment of the present application;

[0015] Figure 5 is a structural schematic diagram of a depth information optimization device according to an embodiment of the present application;

[0016] Figure 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

[0017] Figure 7It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0019] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0020] The depth information optimization method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0021] like Figure 1 As shown, an embodiment of the present application provides a depth information optimization method, which can be executed by an electronic device. In other words, the method can be executed by software or hardware installed in the electronic device. The method includes the following steps:

[0022] Step 110: Determine a first depth matrix, where the first depth matrix is ​​a mixing depth matrix of the TOF camera under a first frequency signal and a second frequency signal, and the frequency of the first frequency signal is higher than the frequency of the second frequency signal.

[0023] In the embodiment of the present application, the TOF camera can perform dual-frequency ranging, that is, the TOF camera can emit signals of two frequencies, and measure the same scene using two different frequency signals to obtain two measurement results under two different frequency signals, and combine the two measurement results to determine the final distance to obtain a mixed depth matrix. The first depth matrix is ​​the mixed depth matrix obtained by the TOF camera using the first frequency signal and the second frequency signal for dual-frequency ranging.

[0024] Since the first depth matrix is ​​a mixed depth matrix of the TOF camera at the first frequency signal and the second frequency signal, it is more real and reliable than the measurement result of the TOF camera at a single frequency.

[0025] Step 120: Obtain a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera.

[0026] The high-frequency signal has a short wavelength and can better highlight detail information. The frequency of the first frequency signal is higher than that of the second frequency signal, so a first confidence matrix under the first frequency signal is obtained.

[0027] In the embodiment of the present application, the TOF camera transmits two signals with a frequency of 20M and a frequency of 100M as an example. The frequency of the 100M frequency signal is higher and the wavelength is shorter. The data collected under the 100M frequency signal has more detailed information, so the first confidence matrix under the 100M signal is obtained.

[0028] Obtaining an initial intrinsic parameter matrix of the TOF camera. The initial intrinsic parameter matrix of the TOF camera can be obtained based on the intrinsic parameters of the TOF camera, and the intrinsic parameters of the TOF camera are related to the camera's own characteristics, such as image center, pixel focal length, etc.

[0029] Step 130: Optimize the first depth matrix based on the first confidence matrix and the initial intrinsic parameter matrix.

[0030] By using the confidence information under high-frequency signals and the initial internal reference information of the TOF camera, the depth information is optimized to achieve depth area guidance.

[0031] Therefore, an embodiment of the present application provides a depth information optimization method, by determining a first depth matrix, where the first depth matrix is ​​a mixed depth matrix of a time-of-flight TOF camera under a first frequency signal and a second frequency signal, and the frequency of the first frequency signal is higher than the frequency of the second frequency signal; obtaining a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera; based on the first confidence matrix and the initial intrinsic parameter matrix, optimizing the first depth matrix, being able to use the confidence under the high-frequency signal as a guide for the depth information, optimizing the dual-frequency depth calculation method of the TOF camera, highlighting the detail information, improving the quality of the depth map, and solving the problem of low quality of the output depth map.

[0032] like Figure 2 As shown, an embodiment of the present application provides a depth information optimization method, which can be executed by an electronic device. In other words, the method can be executed by software or hardware installed in the electronic device. The method includes the following steps:

[0033] Step 210: Determine a first depth matrix, where the first depth matrix is ​​a mixing depth matrix of a time-of-flight TOF camera under a first frequency signal and a second frequency signal, where the frequency of the first frequency signal is higher than the frequency of the second frequency signal.

[0034] In the embodiment of the present application, step 210 includes the following steps:

[0035] Step 211: Determine a first distance matrix and a first confidence matrix under the first frequency signal according to the raw data collected by the TOF camera under the first frequency signal.

[0036] Step 212: Determine a second distance matrix and a second confidence matrix under the second frequency signal according to the raw data collected by the TOF camera under the second frequency signal.

[0037] Take the TOF camera transmitting two signals with frequencies of 20M and 100M, each frequency signal has a distribution of four phases (0°, 90°, 180°, 270°), and the resolution of the acquired depth map is 240*180 as an example for explanation.

[0038] The raw data collected by the TOF camera can be divided into 8 matrices of 240*180, which are x0, x1, x2, x3, x4, x5, x6, and x7, respectively, where x0 to x3 are the raw data of different phases (0°, 90°, 180°, 270°) acquired under the 20M frequency signal, and x4 to x7 are the raw data of different phases acquired under the 100M frequency. In other words, the raw data collected by the TOF camera under the 20M frequency signal are x0 to x3, and the distance matrix and confidence matrix under the 20M frequency signal are determined based on x0 to x3.

[0039] In one implementation, the distance matrix and the confidence matrix under the 20M frequency signal can be calculated based on the original data according to the following formula:

[0040] temp1 20M =x0-x2

[0041] temp2 20M =x3-x1

[0042] confidence 20M =abs(temp1 20M )+abs(temp2 20M )

[0043] rad 20M =arctan(temp2 20M / temp1 20M )

[0044] distance 20M =(7.5*rad 20M / (2*π))

[0045] Among them, confidence 20M Represents the confidence matrix under 20M frequency signal, distance 20M Represents the distance matrix under 20M frequency signal.

[0046] Similarly, the distance matrix and confidence matrix under the 100M frequency signal can be determined based on the original data x4 to x7 collected by the TOF camera under the 100M frequency signal, which will not be described in detail here.

[0047] Step 213: Determine the first depth matrix based on the first distance matrix, the first confidence matrix, the second distance matrix and the second confidence matrix.

[0048] According to steps 211-212, the first distance matrix distance under the 100M frequency signal can be obtained. 100M , the first confidence matrixconfidence 100M , and the second distance matrix distance under 20M frequency signal 20M , the second confidence matrixconfidence 20M .

[0049] In one implementation, based on the first distance matrix distance 100M , the first confidence matrixconfidence 100M , the second distance matrix distance 20M and the second confidence matrix 20M According to the Chinese Residue Theory, the mixing depth matrix, that is, the first depth matrix depth, can be calculated.

[0050] Step 220: Obtain a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera.

[0051] Step 230: Optimize the first depth matrix based on the first confidence matrix and the initial intrinsic parameter matrix.

[0052] Steps 220-230 may be performed using Figure 1 The description of steps 120 - 130 of the embodiment will not be repeated here.

[0053] Therefore, an embodiment of the present application provides a depth information optimization method, by determining a first depth matrix, where the first depth matrix is ​​a mixed depth matrix of a time-of-flight TOF camera under a first frequency signal and a second frequency signal, and the frequency of the first frequency signal is higher than the frequency of the second frequency signal; obtaining a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera; based on the first confidence matrix and the initial intrinsic parameter matrix, optimizing the first depth matrix, being able to use the confidence under the high-frequency signal as a guide for the depth information, optimizing the dual-frequency depth calculation method of the TOF camera, highlighting the detail information, improving the quality of the depth map, and solving the problem of low quality of the output depth map.

[0054] Therefore, an embodiment of the present application provides a depth information optimization method, which determines a first distance matrix and a first confidence matrix under the first frequency signal according to the original data collected by the TOF camera under the first frequency signal; determines a second distance matrix and a second confidence matrix under the second frequency signal according to the original data collected by the TOF camera under the second frequency signal; determines the first depth matrix based on the first distance matrix, the first confidence matrix, the second distance matrix and the second confidence matrix, and can determine a mixed depth matrix based on the original data collected by the TOF camera under the two frequency signals, so that the depth information is more real and reliable.

[0055] like Figure 3 As shown, an embodiment of the present application provides a depth information optimization method, which can be executed by an electronic device. In other words, the method can be executed by software or hardware installed in the electronic device. The method includes the following steps:

[0056] Step 310: Determine a first depth matrix, where the first depth matrix is ​​a mixing depth matrix of a time-of-flight TOF camera under a first frequency signal and a second frequency signal, where the frequency of the first frequency signal is higher than the frequency of the second frequency signal.

[0057] Step 310 may be performed by Figure 1 The description of step 110 of the embodiment can also be adopted Figure 2 The description of steps 211-213 of the embodiment will not be repeated here.

[0058] Step 320: Obtain a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera.

[0059] Step 320 may be performed by Figure 1 The description of step 120 of the embodiment will not be repeated here.

[0060] Step 330: Optimize the first depth matrix based on the first confidence matrix and the initial intrinsic parameter matrix.

[0061] In the embodiment of the present application, step 330 includes the following steps:

[0062] Step 331: Determine a first confidence region matrix based on the first confidence matrix and the initial internal parameter matrix.

[0063] In one implementation, based on the first confidence matrix confidence 100M The first confidence region matrix confidence is calculated according to the following formula: temp :

[0064] confidence temp =confidence 100M 2 *temp 2

[0065] Step 332: perform a dot product of the first confidence region matrix and the first depth matrix to obtain a dot product matrix.

[0066] In one implementation, based on the first confidence region matrix confidence temp The dot product matrix CD is calculated with the first depth matrix depth according to the following formula:

[0067] CD=depth.*confidence temp

[0068] Step 333: Based on the dot product matrix, remove invalid elements in the first confidence region matrix and mark connected regions to obtain a second confidence region matrix.

[0069] Points that are too far or too close have smaller CD values, and such points are considered invalid. That is, points with CD values ​​less than the first threshold are considered invalid. temp The position element corresponding to the point is set to 0 to remove the invalid point.

[0070] Mark the connected domain of the confidence region after removing invalid points, set the value of the marked position element in the confidence region to 1, and obtain the second confidence region matrix confidence _Area .

[0071] Step 334: perform a dot multiplication of the second confidence region matrix and the first depth matrix to obtain a second depth matrix.

[0072] In one implementation, based on the second confidence region matrix confidence _Area The second depth matrix dep is calculated using the following formula with the first depth matrix depth temp :

[0073] dep temp =depth.*Confidence_Area

[0074] Therefore, an embodiment of the present application provides a depth information optimization method, by determining a first depth matrix, where the first depth matrix is ​​a mixed depth matrix of a time-of-flight TOF camera under a first frequency signal and a second frequency signal, and the frequency of the first frequency signal is higher than the frequency of the second frequency signal; obtaining a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera; based on the first confidence matrix and the initial intrinsic parameter matrix, optimizing the first depth matrix, being able to use the confidence under the high-frequency signal as a guide for the depth information, optimizing the dual-frequency depth calculation method of the TOF camera, highlighting the detail information, improving the quality of the depth map, and solving the problem of low quality of the output depth map.

[0075] Therefore, an embodiment of the present application provides a depth information optimization method, which determines a first confidence area matrix based on the first confidence matrix and the initial intrinsic parameter matrix; multiplies the first confidence area matrix by the first depth matrix to obtain a dot product matrix; based on the dot product matrix, removes invalid elements in the first confidence area matrix and marks connected domains to obtain a second confidence area matrix; multiplies the second confidence area matrix by the first depth matrix to obtain a second depth matrix, and can determine invalid points based on the dot product matrix of the first confidence area matrix and the first depth matrix, thereby removing invalid points in the depth map.

[0076] like Figure 4 As shown, an embodiment of the present application provides a depth information optimization method, which can be executed by an electronic device. In other words, the method can be executed by software or hardware installed in the electronic device. The method includes the following steps:

[0077] Step 410: Determine a first depth matrix, where the first depth matrix is ​​a mixing depth matrix of a TOF camera under a first frequency signal and a second frequency signal, where the frequency of the first frequency signal is higher than the frequency of the second frequency signal.

[0078] Step 410 may be performed using Figure 1 The description of step 110 of the embodiment can also be adopted Figure 2The description of steps 211-213 of the embodiment will not be repeated here.

[0079] Step 420: Obtain a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera.

[0080] Step 420 may be performed by Figure 1 The description of step 120 of the embodiment will not be repeated here.

[0081] Step 430: Optimize the first depth matrix based on the first confidence matrix and the initial intrinsic parameter matrix.

[0082] Step 430 may include the following steps:

[0083] Step 431: Determine a first confidence region matrix based on the first confidence matrix and the initial internal parameter matrix.

[0084] Step 432: perform a dot product of the first confidence region matrix and the first depth matrix to obtain a dot product matrix.

[0085] Step 433: Based on the dot product matrix, remove invalid elements in the first confidence region matrix and mark connected regions to obtain a second confidence region matrix.

[0086] Step 434: perform a dot multiplication of the second confidence region matrix and the first depth matrix to obtain a second depth matrix.

[0087] Steps 431-434 can be performed using Figure 3 The description of steps 331-334 of the embodiment will not be repeated here.

[0088] In the embodiment of the present application, step 430 further includes the following steps:

[0089] Step 435: Perform a histogram equalization operation based on the first confidence matrix and the initial internal parameter matrix to obtain a third confidence matrix.

[0090] In one implementation, based on the first confidence matrix confidence 100M The initial internal parameter matrix temp is used to obtain a histogram equalization matrix confidence according to the following formula: _balance :

[0091] Confidence_balance=confidence 100M *temp 2

[0092] For the above histogram equalization matrix confidence_balance Perform histogram equalization. Specifically, count the confidence matrix to be histogram equalized _balance The number of each gray value in the image, calculate the probability of each gray value appearing, and complete the new gray value mapping. The function of the mapping method is as follows:

[0093]

[0094] Among them, n represents the number of all pixels, n j represents the number of pixels with gray value j, k represents the gray value, S k Represents the new gray level. After the above histogram equalization, the third confidence matrix is ​​obtained.

[0095] Step 436: perform kernel density estimation on the third confidence matrix to determine the number of minimum values.

[0096] In one implementation, a Gaussian kernel function is used to perform kernel density estimation on the third confidence matrix. Minimum values ​​can be obtained based on the kernel density estimation result, and the number of minimum values ​​is determined.

[0097] Then, the third confidence matrix may be classified into intervals according to the minimum value.

[0098] Step 437: When the number of the minimum values ​​is greater than zero, perform a background filtering operation on the first depth matrix to obtain a valid region matrix of the first depth matrix.

[0099] When the number of the minimum values ​​is greater than zero, it indicates that the third confidence matrix has more than one interval classification.

[0100] In one implementation, the third confidence matrix is ​​connected to mark the domain, and the same number is assigned to the regions belonging to the same connected domain. The regions marked with numbers are all valid points, which are recorded as bw_target matrix. Correspondingly, the bw_background matrix is ​​recorded as:

[0101] bw_background=1-bw_target

[0102] A background filter operation is performed on the first depth matrix depth, and the result is recorded as depth_filter. The background filter operation includes a median filter operation.

[0103] In one implementation, the background effective matrix background, the foreground effective matrix target and the effective area matrix depth of the first depth matrix are calculated according to the following formula: valid :

[0104] background=bw_background.*depth_filter

[0105] target = bw_target.*depth

[0106] depthv alid =background+target

[0107] In one implementation, after performing background filtering operation on the first depth matrix, bilateral filtering operation may be performed on the first depth matrix depth and depth_filter respectively, thereby maintaining the edge and achieving a denoising effect at the same time.

[0108] Step 438: Perform a morphological opening operation on the valid region matrix of the first depth matrix based on the second confidence region matrix to obtain a third depth matrix.

[0109] In order to eliminate noise to the maximum extent, a morphological opening operation is performed on the valid area matrix depthvalid of the first depth matrix. In one implementation, the third depth matrix depthend is calculated according to the following formula.

[0110] depth end =depth valid .*Confidence_Area

[0111] Step 439: When the number of the minimum values ​​is equal to zero, perform a median filter operation, a bilateral filter operation, and a morphological opening operation on the second depth matrix to obtain a fourth depth matrix.

[0112] When the number of minimum values ​​is equal to 0, it means that the confidence map has only one interval classification. In this case, the second depth matrix dep temp Median filtering and bilateral filtering are performed in sequence, and in order to remove noise as much as possible, the second depth matrix dep temp Perform morphological opening operation to obtain the fourth depth matrix depth end .

[0113] The noise of the depth map obtained after the median filtering operation, the bilateral filtering operation and the body opening operation optimization is significantly reduced and has more details.

[0114] Therefore, an embodiment of the present application provides a depth information optimization method, by determining a first depth matrix, where the first depth matrix is ​​a mixed depth matrix of a time-of-flight TOF camera under a first frequency signal and a second frequency signal, and the frequency of the first frequency signal is higher than the frequency of the second frequency signal; obtaining a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera; based on the first confidence matrix and the initial intrinsic parameter matrix, optimizing the first depth matrix, being able to use the confidence under the high-frequency signal as a guide for the depth information, optimizing the dual-frequency depth calculation method of the TOF camera, highlighting the detail information, improving the quality of the depth map, and solving the problem of low quality of the output depth map.

[0115] Therefore, a method for optimizing depth information provided by an embodiment of the present application performs a histogram equalization operation based on the first confidence matrix and the initial internal parameter matrix to obtain a third confidence matrix; performs kernel density estimation on the third confidence matrix to determine the number of minimum values; when the number of minimum values ​​is greater than zero, performs median filtering and bilateral filtering operations on the first depth matrix to obtain the effective area matrix of the first depth matrix; performs a morphological opening operation on the effective area matrix of the first depth matrix based on the second confidence area matrix to obtain a third depth matrix. When the number of minimum values ​​is equal to zero, performs a median filtering operation, a bilateral filtering operation, and a morphological opening operation on the second depth matrix to obtain a fourth depth matrix, which can reduce noise by performing a histogram equalization operation on the confidence matrix, and achieve the effect of further denoising while maintaining edge details based on the median filtering operation, the bilateral filtering operation, and the morphological opening operation.

[0116] It should be noted that the depth information optimization method provided in the embodiment of the present application can be executed by a depth information optimization device, or a module in the depth information optimization device for executing the depth information optimization method. In the embodiment of the present application, the depth information optimization device provided in the embodiment of the present application is described by taking the execution of the depth information optimization method by the depth information optimization device as an example.

[0117] Figure 5 Schematic diagram of the structure of the depth information optimization device according to the embodiment of the present application. Figure 5 As shown, the depth information optimization device 500 includes: a determination module 510, an acquisition module 520 and an optimization module 530.

[0118] A determination module 510 is used to determine a first depth matrix, where the first depth matrix is ​​a mixed depth matrix of a time-of-flight TOF camera under a first frequency signal and a second frequency signal, and the frequency of the first frequency signal is higher than the frequency of the second frequency signal; an acquisition module 520 is used to acquire a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera; and an optimization module 530 is used to optimize the first depth matrix based on the first confidence matrix and the initial intrinsic parameter matrix.

[0119] In one implementation, the determination module 510 is used to: determine a first distance matrix and a first confidence matrix under the first frequency signal based on the original data collected by the TOF camera under the first frequency signal; determine a second distance matrix and a second confidence matrix under the second frequency signal based on the original data collected by the TOF camera under the second frequency signal; and determine the first depth matrix based on the first distance matrix, the first confidence matrix, the second distance matrix, and the second confidence matrix.

[0120] In one implementation, the optimization module 530 is used to: determine a first confidence region matrix based on the first confidence matrix and the initial intrinsic parameter matrix; perform dot product of the first confidence region matrix and the first depth matrix to obtain a dot product matrix; based on the dot product matrix, remove invalid elements in the first confidence region matrix and perform connected domain marking to obtain a second confidence region matrix; perform dot product of the second confidence region matrix and the first depth matrix to obtain a second depth matrix.

[0121] In one implementation, the optimization module 530 is also used to: perform a histogram equalization operation based on the first confidence matrix and the initial internal parameter matrix to obtain a third confidence matrix; perform kernel density estimation on the third confidence matrix to determine the number of minimum values; when the number of minimum values ​​is greater than zero, perform a background filtering operation on the first depth matrix to obtain a valid area matrix of the first depth matrix; perform a morphological opening operation on the valid area matrix of the first depth matrix based on the second confidence area matrix to obtain a third depth matrix.

[0122] In one implementation, the optimization module 530 is further configured to: when the number of the minimum values ​​is greater than zero, perform background filtering and bilateral filtering on the first depth matrix to obtain a valid region matrix of the first depth matrix.

[0123] In one implementation, after performing kernel density estimation on the third confidence matrix and determining the number of minimum values, the optimization module 530 is also used to: when the number of minimum values ​​is equal to zero, perform median filtering, bilateral filtering and morphological opening operations on the second depth matrix to obtain a fourth depth matrix.

[0124] Therefore, an embodiment of the present application provides a depth information optimization device, which determines a first depth matrix, where the first depth matrix is ​​a mixed depth matrix of a time-of-flight TOF camera under a first frequency signal and a second frequency signal, and the frequency of the first frequency signal is higher than the frequency of the second frequency signal; obtains a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera; based on the first confidence matrix and the initial intrinsic parameter matrix, optimizes the first depth matrix, and can use the confidence under the high-frequency signal to guide the depth information, optimize the dual-frequency depth calculation method of the TOF camera, highlight the detail information, improve the quality of the depth map, and solve the problem of low quality of the output depth map.

[0125] The deep information optimization device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0126] The depth information optimization device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0127] The depth information optimization device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0128] Alternatively, if Figure 6As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601, a memory 602, and a program or instruction stored in the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, each process of the above-mentioned depth information optimization method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0129] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0130] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0131] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0132] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0133] Among them, the processor 710 is used to determine a first depth matrix, which is a mixed depth matrix of the time-of-flight TOF camera under a first frequency signal and a second frequency signal, and the frequency of the first frequency signal is higher than the frequency of the second frequency signal; obtain a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera; based on the first confidence matrix and the initial intrinsic parameter matrix, optimize the first depth matrix.

[0134] In one implementation, the processor 710 is further used to determine a first distance matrix and a first confidence matrix under the first frequency signal based on the original data collected by the TOF camera under the first frequency signal; determine a second distance matrix and a second confidence matrix under the second frequency signal based on the original data collected by the TOF camera under the second frequency signal; and determine the first depth matrix based on the first distance matrix, the first confidence matrix, the second distance matrix, and the second confidence matrix.

[0135] In one implementation, the processor 710 is further used to determine a first confidence region matrix based on the first confidence matrix and the initial intrinsic parameter matrix; perform dot product of the first confidence region matrix with the first depth matrix to obtain a dot product matrix; based on the dot product matrix, remove invalid elements in the first confidence region matrix and perform connected domain marking to obtain a second confidence region matrix; perform dot product of the second confidence region matrix with the first depth matrix to obtain a second depth matrix.

[0136] In one implementation, the processor 710 is further used to perform a histogram equalization operation based on the first confidence matrix and the initial intrinsic parameter matrix to obtain a third confidence matrix; perform kernel density estimation on the third confidence matrix to determine the number of minimum values; when the number of minimum values ​​is greater than zero, perform a background filtering operation on the first depth matrix to obtain a valid area matrix of the first depth matrix; and perform a morphological opening operation on the valid area matrix of the first depth matrix based on the second confidence area matrix to obtain a third depth matrix.

[0137] In one implementation, the processor 710 is further configured to, when the number of the minimum values ​​is greater than zero, perform background filtering and bilateral filtering on the first depth matrix to obtain a valid region matrix of the first depth matrix.

[0138] In one implementation, the processor 710 is further used to perform a median filter operation, a bilateral filter operation, and a morphological opening operation on the second depth matrix after performing a kernel density estimation on the third confidence matrix and determining the number of minimum values ​​when the number of minimum values ​​is equal to zero, so as to obtain a fourth depth matrix.

[0139] It should be understood that in the embodiment of the present application, the input unit 704 may include a graphics processor (Graphics Processing Unit, GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include but are not limited to a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 709 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 can integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and applications, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 710.

[0140] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned depth information optimization method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0141] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0142] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned deep information optimization method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0143] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0144] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0146] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A depth information optimization method, characterized in that: The method comprises: Determine a first depth matrix, where the first depth matrix is ​​a mixing depth matrix of a time-of-flight TOF camera under a first frequency signal and a second frequency signal, where a frequency of the first frequency signal is higher than a frequency of the second frequency signal; Acquire a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera; Optimizing the first depth matrix based on the first confidence matrix and the initial intrinsic parameter matrix; The optimizing the first depth matrix based on the first confidence matrix and the initial intrinsic parameter matrix includes: Determine a first confidence region matrix based on the first confidence matrix and the initial internal parameter matrix; Perform a dot product of the first confidence region matrix and the first depth matrix to obtain a dot product matrix; Based on the dot product matrix, invalid elements in the first confidence region matrix are removed and connected domains are marked to obtain a second confidence region matrix; The second confidence region matrix is ​​dot-multiplied by the first depth matrix to obtain a second depth matrix.

2. The method according to claim 1, characterized in that Determining the first depth matrix comprises: Determine a first distance matrix and a first confidence matrix under the first frequency signal according to the raw data collected by the TOF camera under the first frequency signal; Determine a second distance matrix and a second confidence matrix under the second frequency signal according to the raw data collected by the TOF camera under the second frequency signal; The first depth matrix is ​​determined based on the first distance matrix, the first confidence matrix, the second distance matrix, and the second confidence matrix.

3. The method according to claim 1, characterized in that The optimizing the first depth matrix based on the first confidence matrix and the initial intrinsic parameter matrix further includes: Based on the first confidence matrix and the initial internal parameter matrix, a histogram equalization operation is performed to obtain a third confidence matrix; Performing kernel density estimation on the third confidence matrix to determine the number of minimum values; When the number of the minimum values ​​is greater than zero, performing a background filtering operation on the first depth matrix to obtain a valid region matrix of the first depth matrix; A morphological opening operation is performed on the valid region matrix of the first depth matrix based on the second confidence region matrix to obtain a third depth matrix.

4. The method according to claim 3, characterized in that When the number of the minimum values ​​is greater than zero, performing a background filtering operation on the first depth matrix to obtain a valid area matrix of the first depth matrix includes: When the number of the minimum values ​​is greater than zero, background filtering operation and bilateral filtering operation are performed on the first depth matrix to obtain a valid area matrix of the first depth matrix.

5. The method according to claim 3, characterized in that: After performing kernel density estimation on the third confidence matrix and determining the number of minimum values, the method further includes: When the number of the minimum values ​​is equal to zero, a median filtering operation, a bilateral filtering operation and a morphological opening operation are performed on the second depth matrix to obtain a fourth depth matrix.

6. A depth information optimization device, characterized in that: The device comprises: A determination module, configured to determine a first depth matrix, where the first depth matrix is ​​a mixed depth matrix of a time-of-flight TOF camera under a first frequency signal and a second frequency signal, where the frequency of the first frequency signal is higher than the frequency of the second frequency signal; An acquisition module, used to acquire a first confidence matrix under the first frequency signal and an initial intrinsic parameter matrix of the TOF camera; An optimization module, configured to optimize the first depth matrix based on the first confidence matrix and the initial intrinsic parameter matrix; Among them, the optimization module is used to determine a first confidence region matrix based on the first confidence matrix and the initial intrinsic parameter matrix; perform dot product of the first confidence region matrix and the first depth matrix to obtain a dot product matrix; based on the dot product matrix, remove invalid elements in the first confidence region matrix and mark connected domains to obtain a second confidence region matrix; perform dot product of the second confidence region matrix and the first depth matrix to obtain a second depth matrix.

7. The device according to claim 6, characterized in that The determination module is used for: Determine a first distance matrix and a first confidence matrix under the first frequency signal according to the raw data collected by the TOF camera under the first frequency signal; Determine a second distance matrix and a second confidence matrix under the second frequency signal according to the raw data collected by the TOF camera under the second frequency signal; The first depth matrix is ​​determined based on the first distance matrix, the first confidence matrix, the second distance matrix, and the second confidence matrix.

8. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method according to any one of claims 1 to 5.

9. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images