Depth measurement method, device and terminal equipment

Abstract

The present application is applicable to the field of measurement technology, and provides a depth measurement method, device and terminal equipment, the method comprising: respectively collecting depth images of the current scene at at least two measurement frequencies, the at least two measurement frequencies at least including a first frequency and a second frequency, the first frequency being greater than the second frequency, and the depth image comprising a plurality of pixels; determining a plurality of candidate fusion depths for each pixel, and determining a relative depth for each pixel; identifying the fusion depth of the target pixel according to the plurality of candidate fusion depths and relative depths of the target pixel, the target pixel being any one of the plurality of pixels; generating a target depth image of the current scene based on the fusion depths of the plurality of target pixels. The above method takes into account the changes in the front-to-back relationship between pixels in the fusion calculation, can effectively protect the original depth logic of the scene, and improves the accuracy of depth measurement.

Description

Technical Field

[0001] The present application belongs to the field of measurement technology, and in particular, relates to a depth measurement method, device and terminal equipment. Background Art

[0002] A Time of Flight (TOF) camera is a detector that measures the depth of an object (i.e., the distance relative to the TOF camera) by obtaining the flight time of the detection wave. A TOF camera can detect the depth of objects in a scene and output a depth map. The value of each pixel in the depth map is the depth of that point.

[0003] The measurement range of the TOF camera is determined by the modulation frequency of its detection wave, which is equal to half the speed of light divided by the modulation frequency. For example, when using the measurement mode of 20 MHz modulation frequency, its range is 7.5 meters, and when using the measurement mode of 60 MHz modulation frequency, its range is 2.5 meters. When the depth of the object exceeds the range corresponding to a certain frequency, the output depth of the measurement mode is the "folded" depth, that is, the true depth is modulo the remainder of the range. For example, in the 60 MHz measurement mode, an object at 3.5 meters will generate 1 fold because the depth exceeds the 2.5 meter range, and the output depth measurement value is 1 meter; accordingly, if the output depth measurement value is 1 meter, then the true depth value of the object may be 1 meter, or 3.5 meters (folded once), or 6 meters (folded twice), and so on. Therefore, when restoring the true depth value of the object based on the depth measurement value output by the TOF camera, if the number of folds is judged incorrectly, the final true depth value of the object will also be wrong, which will seriously affect the accuracy of the depth measurement. Summary of the invention

[0004] The embodiments of the present application provide a depth measurement method, apparatus and terminal device, which can improve the accuracy of depth measurement.

[0005] In a first aspect, an embodiment of the present application provides a depth measurement method, including:

[0006] The depth images of the current scene at at least two measurement frequencies are collected respectively, and the at least two measurement frequencies include at least a first frequency and a second frequency, the first frequency is greater than the second frequency, and accordingly, the measurement range of the first frequency is smaller than the measurement range of the second frequency. For each pixel in the depth image, multiple candidate fusion depths of each pixel and the relative depth of the pixel can be determined, so that the fusion depth of the target pixel can be identified based on the multiple candidate fusion depths and relative depths of the target pixel, and the target pixel is any one of the multiple pixels. After identifying the fusion depth of each target pixel, the target depth image of the current scene can be generated based on the fusion depths of the multiple target pixels.

[0007] The embodiment of the present application combines the relative depth information of each pixel point and considers the changes in the front-back relationship between the pixels in the fusion calculation, thereby effectively protecting the original depth logic of the scene and improving the accuracy of the depth measurement.

[0008] In a possible implementation of the first aspect, multiple candidate fusion depths for each pixel can be obtained by multiple folding numbers of high-frequency measurement values. Therefore, a first depth value of each pixel at a first frequency, that is, a high-frequency measurement value, can be obtained respectively, and then multiple candidate fusion depths for each pixel are calculated based on the first depth value.

[0009] In a possible implementation of the first aspect, when calculating multiple candidate fusion depths for each pixel point based on the first depth value, the measurement range corresponding to the first frequency, that is, the measurement range corresponding to the high-frequency modulation frequency, can be first determined, and then the sum of the first depth value of each pixel point and multiple non-negative integer multiple values ​​of the measurement range corresponding to the first frequency is calculated respectively, and the sum of the first depth value and the multiple non-negative integer multiple values ​​is used as the multiple candidate fusion depths for each pixel point. The above multiple non-negative integers may include 0, 1, 2, and so on.

[0010] In a possible implementation of the first aspect, the relative depth of each pixel can be determined by a low-frequency measurement value of the pixel. Therefore, the second depth value of each pixel at the low-frequency modulation frequency, that is, the second frequency, can be obtained respectively, and then each pixel is sorted in the order of the second depth value from small to large, and by determining the initial sorting number of each pixel after sorting, the initial sorting number can be used as the relative depth of the corresponding pixel.

[0011] In a possible implementation manner of the first aspect, the relative position change between each candidate fusion depth of the target pixel and the relative depth of the target pixel can be determined respectively, and then the candidate fusion depth corresponding to the minimum value of the relative position change is identified as the fusion depth of the target pixel.

[0012] In a possible implementation of the first aspect, the relative position change between each candidate fusion depth of the target pixel and the relative depth of the target pixel can be determined by inserting each candidate fusion depth of the target pixel into a sequence sorted according to the second depth value, and by calculating the difference in sequence number value between the sorting number of each candidate fusion depth in the sequence and the initial sorting number of the target pixel, the difference in sequence number value can be used as the above-mentioned relative position change.

[0013] In a possible implementation of the first aspect, the relative depth of each pixel point can also be determined by the difference between the distances between the candidate fusion depths corresponding to the low-frequency measurement value and the high-frequency measurement value. Therefore, the second depth value of each pixel point at the second frequency can be obtained respectively, and the distance difference between each candidate fusion depth of each pixel point and the second depth value can be calculated, and the minimum value of the distance difference can be extracted as the minimum fusion distance of the corresponding pixel point, and then the relative depth of each pixel point can be determined according to the minimum fusion distance.

[0014] In a possible implementation of the first aspect, the relative depth of each pixel is determined according to the minimum fusion distance, and all pixels can be first divided into reference pixels and non-reference pixels according to the minimum fusion distance. Among them, the pixels whose minimum fusion distance is less than a preset threshold can be identified as reference pixels, and the pixels whose minimum fusion distance is greater than or equal to the preset threshold can be identified as non-reference pixels. Then, the reference pixels are sorted in the order of the second depth value from small to large, and the initial sorting number of each reference pixel after sorting is determined, and the initial sorting number can be used as the relative depth of the reference pixel. For non-reference pixels, the second depth value of the non-reference pixel can be inserted into the sorting sequence of the reference pixel, and the sorting number can be used as the relative depth of the non-reference pixel by determining the sorting number of the second depth value of the non-reference pixel in the sorting sequence of the reference pixel.

[0015] In a possible implementation of the first aspect, the target pixel may include a reference pixel and a non-reference pixel. Therefore, when identifying the fusion depth of the target pixel, different judgment methods may be used depending on whether it is a reference pixel. Specifically, for a reference pixel, the candidate fusion depth corresponding to the minimum fusion distance of the reference pixel may be identified as its fusion depth; and for a non-reference pixel, the relative position change between each candidate fusion depth of the non-reference pixel and the relative depth of the non-reference pixel may be determined respectively, and then the candidate fusion depth corresponding to the minimum value of the relative position change may be identified as the fusion depth of the non-reference pixel.

[0016] In a second aspect, an embodiment of the present application provides a depth measurement device, including:

[0017] A depth image acquisition module, used to respectively acquire depth images of the current scene at at least two measurement frequencies, wherein the at least two measurement frequencies include at least a first frequency and a second frequency, wherein the first frequency is greater than the second frequency, and the depth image includes a plurality of pixels;

[0018] a candidate fusion depth determination module, used to determine multiple candidate fusion depths for each pixel; and

[0019] A relative depth determination module, used to determine the relative depth of each pixel;

[0020] A fusion depth recognition module, used to identify the fusion depth of a target pixel point according to a plurality of candidate fusion depths and a relative depth of the target pixel point, wherein the target pixel point is any one of the plurality of pixel points;

[0021] The target depth image generation module is used to generate a target depth image of the current scene based on the fused depth of multiple target pixels.

[0022] In a possible implementation manner of the second aspect, the candidate fusion depth determination module may specifically include the following submodules:

[0023] A first depth value acquisition submodule, used to respectively acquire a first depth value of each pixel point at a first frequency;

[0024] The candidate fusion depth calculation submodule is used to calculate multiple candidate fusion depths of each pixel point according to the first depth value.

[0025] In a possible implementation manner of the second aspect, the candidate fusion depth calculation submodule may specifically include the following units:

[0026] a measurement range determining unit, configured to determine a measurement range corresponding to the first frequency;

[0027] The candidate fusion depth calculation unit is used to respectively calculate the sum of the first depth value of each pixel point and multiple non-negative integer multiple values ​​of the measurement range corresponding to the first frequency, and use the sum of the first depth value and the multiple non-negative integer multiple values ​​as the multiple candidate fusion depths of each pixel point.

[0028] In a possible implementation manner of the second aspect, the relative depth determination module may specifically include the following submodules:

[0029] A second depth value acquisition submodule, used to respectively acquire a second depth value of each pixel point at a second frequency;

[0030] A pixel sorting submodule, used to sort each pixel in the order of the second depth value from small to large;

[0031] The first relative depth determination submodule is used to determine an initial sorting number of each pixel after sorting, and use the initial sorting number as the relative depth of the pixel.

[0032] In a possible implementation manner of the second aspect, the fusion depth recognition module may specifically include the following submodules:

[0033] A relative position change determination submodule is used to respectively determine the relative position change between each candidate fusion depth of the target pixel and the relative depth of the target pixel;

[0034] The fusion depth recognition submodule is used to recognize the candidate fusion depth corresponding to the minimum value of the relative position change as the fusion depth of the target pixel point.

[0035] In a possible implementation manner of the second aspect, the relative position change amount determining submodule may specifically include the following units:

[0036] A candidate fused depth insertion unit, configured to insert each candidate fused depth of the target pixel into a sequence sorted according to the second depth value;

[0037] The relative position change calculation unit is used to calculate the difference between the sorting number of each candidate fusion depth in the sequence and the initial sorting number of the target pixel point, and use the difference in the number value as the relative position change.

[0038] In a possible implementation manner of the second aspect, the relative depth determination module may further include the following submodules:

[0039] A distance difference calculation submodule, used to calculate the distance difference between each candidate fusion depth of each pixel and the second depth value;

[0040] A minimum fusion distance extraction submodule is used to extract the minimum value of the distance difference as the minimum fusion distance of the corresponding pixel point;

[0041] The second relative depth determination submodule is used to determine the relative depth of each pixel according to the minimum fusion distance.

[0042] In a possible implementation manner of the second aspect, the second relative depth determination submodule may specifically include the following units:

[0043] A reference pixel point identification unit, used to identify a pixel point whose minimum fusion distance is less than a preset threshold as a reference pixel point;

[0044] A reference pixel sorting unit, configured to sort the reference pixels in ascending order of the second depth values;

[0045] A reference pixel relative depth determination unit, used to determine an initial sorting number of each reference pixel after sorting, and use the initial sorting number as the relative depth of the reference pixel;

[0046] a non-reference pixel insertion unit, configured to insert, for any non-reference pixel, a second depth value of the non-reference pixel into a sorted sequence of the reference pixels;

[0047] The non-reference pixel relative depth determination unit is used to determine the sorting number of the second depth value of the non-reference pixel in the sorting sequence of the reference pixels, and use the sorting number as the relative depth of the non-reference pixel.

[0048] In a possible implementation manner of the second aspect, the target pixel may include the reference pixel and the non-reference pixel, and the fusion depth recognition module may further include the following submodules:

[0049] A reference pixel fusion depth recognition submodule is used to recognize, for any reference pixel, a candidate fusion depth corresponding to the minimum fusion distance of the reference pixel as the fusion depth of the reference pixel;

[0050] The non-reference pixel fusion depth identification submodule is used to determine, for any non-reference pixel, the relative position change between each candidate fusion depth of the non-reference pixel and the relative depth of the non-reference pixel, and identify the candidate fusion depth corresponding to the minimum value of the relative position change as the fusion depth of the non-reference pixel.

[0051] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the depth measurement method described in any one of the first aspects above when executing the computer program.

[0052] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of a terminal device, the depth measurement method described in any one of the first aspects above is implemented.

[0053] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the depth measurement method described in any one of the above-mentioned first aspects.

[0054] Compared with the prior art, the embodiments of the present application include the following beneficial effects:

[0055] In the embodiment of the present application, by respectively collecting the depth images of the current scene at at least two measurement frequencies, and after determining multiple candidate fusion depths for each pixel and the relative depth of each pixel, the final fusion depth of the pixel can be determined according to the multiple candidate fusion depths and relative depths of each pixel, and then the target depth image of the current scene can be generated. In this embodiment, the folding number of high-frequency measurement values ​​is obtained through the relative depth information provided by the low-frequency measurement values ​​of the pixels, and the changes in the front-to-back relationship between the pixels are considered in the fusion calculation, which can effectively protect the original depth logic of the scene, and can provide the fusion result that best conforms to the original depth logic of the scene, thereby improving the accuracy of the depth measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1(a)-Figure 1(b) is a schematic diagram of a depth measurement method in the prior art;

[0058] Figure 2 is a schematic diagram of an algorithm flow of a depth measurement method provided in an embodiment of the present application;

[0059] Figure 3 is a schematic diagram of the hardware structure of a mobile phone to which the depth measurement method provided in an embodiment of the present application is applicable;

[0060] Figure 4 is a schematic diagram of the software structure of a mobile phone to which the depth measurement method provided in an embodiment of the present application is applicable;

[0061] Figure 5 is a schematic flowchart of the depth measurement method provided in one embodiment of the present application;

[0062] Figure 6 is a schematic diagram of candidate fusion depths provided in an embodiment of the present application;

[0063] Figure 7 is a schematic flowchart of the steps of a depth measurement method provided in another embodiment of the present application;

[0064] Figure 8 is a schematic diagram of candidate fusion depths provided by another embodiment of the present application;

[0065] Fig. 9 is a structural block diagram of a depth measurement device provided by an embodiment of the present application;

[0066] Fig.10 It is a structural diagram of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0067] To facilitate understanding, a common depth measurement method in the prior art is first introduced through a specific example.

[0068] Generally, the resolution of depth measurement by a TOF camera is proportional to the modulation frequency of its detection wave. The higher the frequency, the higher the resolution. At the same time, noise generally interferes less with high-frequency measurement results than low-frequency ones. In order to balance the large range and accuracy, TOF cameras generally use multiple modulation frequency modes for combined measurement, and calculate a fused depth as the final measurement result by applying a fusion algorithm to the measurement values ​​of multiple frequencies. The core of the above fusion algorithm is to determine the number of folding times of a certain frequency measurement value, and then restore its true measurement value.

[0069] The fusion algorithm in the prior art generally determines the number of foldings according to the measured values ​​of each frequency and its possible values ​​according to the proximity principle. The so-called proximity principle means first determining a set of folding numbers so that the distance difference between the possible values ​​of each frequency (i.e., the position after folding) is minimized. The algorithms for determining the combination of folding numbers according to the proximity principle include the traversal method and the fast algorithm based on the Chinese remainder theorem, etc.

[0070] As shown in Figure 1(a), it is a schematic diagram of a depth measurement method in the prior art. In Figure 1, it is assumed that the low-frequency measurement result of 20MHz is not folded, and the possible folding times of the high-frequency measurement result of 60MHz are 0 times, 1 time or 2 times. Among them, the distance difference between the high-frequency possible value of the measurement result folded once and the low-frequency measurement value is the smallest (that is, the distance difference 1 in Figure 1), so it can be determined that the folding times of the high-frequency measurement result is 1. Finally, the final fusion depth can be given according to the possible value of the high-frequency measurement result folded once, or the final fusion depth can be given by weighted averaging the possible value of the high-frequency measurement result folded once and the low-frequency measurement result.

[0071] However, due to the complexity of real-world scenarios, the measurement signal of the TOF camera is easily interfered by factors such as transmitter crosstalk, ambient light noise, and lens stray light, and its frequency measurement values ​​have certain deviations from the actual distance. Especially in scenes with holes or gaps, the signals in the holes and gaps are small and are extremely susceptible to interference, resulting in deviations in the measurement results, especially the low-frequency measurement results have large deviations.

[0072] As shown in Figure 1(b), when the low-frequency measurement value of a pixel in the depth map is small and the distance difference with the unfolded high-frequency measurement value is even smaller (i.e., the distance difference is 0 in Figure 1(b)), the existing fusion algorithm based on the proximity principle will calculate an incorrect high-frequency folding number. At this time, the calculated fusion depth will be much smaller than the actual depth of the object. This error will lead to a serious depth logic error in the depth map where the depth inside the hole is shallower than the hole entrance.

[0073] That is to say, the fusion process in the prior art only utilizes the measurement value of the pixel itself, and does not consider the logical relationship between the pixels. The obtained fusion depth has a large error with the real depth of the object, or is even completely wrong.

[0074] In order to solve the above problems, the core concept of an embodiment of the present application is proposed, which is to first use the measurement results of the TOF camera in the non-folded low-frequency measurement mode to define the relative depth between each pixel on the depth map (that is, the front and back logical relationship of each pixel); then, when calculating the fusion depth, the relative depth change of the pixel is considered to maintain the original depth logic of the scene, thereby correcting the depth logic error of the fusion result in the prior art.

[0075] like Figure 2 As shown, it is a schematic diagram of the algorithm flow of the depth measurement method provided in the embodiment of the present application. Figure 2 The algorithm flow shown in the figure, the depth measurement method of the embodiment of the present application mainly includes the following steps:

[0076] S201, calculating possible fusion positions and corresponding distance differences according to high-frequency and low-frequency measurement results;

[0077] In this step, the high and low frequency modes of the TOF camera can be used to measure the depth of the same scene.

[0078] It should be noted that in order to ensure the validity of the low-frequency measurement results, it should be ensured that in the current scene, the measurement results of the lowest frequency are not folded, that is, the low-frequency range is sufficient to cover the entire scene.

[0079] Generally, the ranges of the 20MHz measurement mode and the 60MHz measurement mode are approximately 7.5 meters and 2.5 meters respectively. For most indoor scenes, 7.5 meters is sufficient to cover all depths in the scene. Therefore, the low-frequency mode in this step can select a modulation frequency of 20MHz, while the high-frequency mode can select a modulation frequency of 60MHz.

[0080] After the measurement is completed, for each pixel point, the fusion position of the depth value corresponding to the different folding numbers of the high-frequency mode and the measured value of the low-frequency mode is calculated.

[0081] For example, for a certain pixel point, the high-frequency measurement value obtained after the measurement is 1 meter, then the depth value corresponding to the pixel point folded 0 times, 1 time or 2 times in the high-frequency mode is 1 meter, or 3.5 meters (folded 1 time), or 6 meters (folded 2 times). Then calculate the fusion position of the above three depth values ​​and the low-frequency mode measurement value, and obtain three fusion positions of the pixel point. That is, the fusion position 0 of the low-frequency mode measurement value and the 1-meter depth value, the fusion position 1 of the low-frequency mode measurement value and the 3.5-meter depth value, and the fusion position 2 of the low-frequency mode measurement value and the 6-meter depth value.

[0082] At the same time, the distance difference between the high-frequency and low-frequency measurement values ​​corresponding to each fusion position should also be calculated. That is, for fusion position 0, the distance difference between the high-frequency measurement value (1 meter) and the low-frequency measurement value of the pixel point is calculated; for fusion position 1, the distance difference between the high-frequency measurement value (3.5 meters) and the low-frequency measurement value of the pixel point is calculated; for fusion position 2, the distance difference between the high-frequency measurement value (6 meters) and the low-frequency measurement value of the pixel point is calculated.

[0083] S202, defining a relative depth of each pixel according to the low-frequency measurement value;

[0084] In this step, the low-frequency measurement results of each pixel in the field of view of the TOF camera (i.e., on the corresponding depth map) can be sorted, and the relative depth of each pixel is defined as the serial number of the pixel in the sorting sequence, which can represent the number of pixels sorted in front of it (including the pixel itself).

[0085] For example, suppose there are 50,000 pixels in the depth map. In the low-frequency measurement mode, each of these 50,000 pixels is measured to obtain a low-frequency measurement value. Then, the 50,000 pixels can be sorted in the order of the low-frequency measurement values. The sorted numbers are 1, 2, 3, ..., 50,000. For any two pixels, the pixel with a relatively large number has a corresponding low-frequency measurement value greater than the pixel with a smaller number.

[0086] Of course, the low-frequency measurement results involved in the sorting can be taken from all pixels, or only from the pixels with a small difference between the high and low frequency distances after fusion in the previous step.

[0087] For example, in the previous step, for fusion position 0, if the calculated distance difference between the high-frequency measurement value (1 meter) and the low-frequency measurement value of the pixel point is less than a certain threshold value, the fusion position 0 can be directly determined as the final fusion position of the pixel point without re-sorting the pixel point. The above threshold value can be determined according to actual needs, and the embodiment of the present application does not limit this.

[0088] S203, calculating the relative depth change of the pixel point at each fusion position relative to the low-frequency measurement value;

[0089] In this step, the relative depth change of the pixel point may mean that, for the pixel point to be evaluated by the depth logic, the ranking of the low-frequency measurement value of the pixel point in the depth sorting sequence may be first found, and then the ranking of a certain fusion position of the pixel point in the depth sorting sequence may be found, and the ranking quantization amount between the two may be calculated.

[0090] For example, for a pixel point to be evaluated by a certain depth logic, assuming that its low-frequency measurement value is found to be 537.1 millimeters (mm), and the ranking number of this measurement value in the depth sorting sequence is 6, then, based on the three fusion positions corresponding to the pixel point calculated in the first step, the positions of these three fusion positions in the above depth sorting sequence are determined respectively.

[0091] It should be noted that the positions of the three fusion positions in the above depth sorting sequence can compare the depth value of each fusion position with the low-frequency measurement value of each pixel in the depth sorting sequence to find the two closest low-frequency measurement values. Among the two closest low-frequency measurement values, one low-frequency measurement value should be the depth value of the fusion position, and the other low-frequency measurement value should be less than the depth value of the fusion position. Then, the sequence number obtained by inserting the depth value of the fusion position between the two low-frequency measurement values ​​is used as the ranking sequence number of the fusion position in the depth sorting sequence.

[0092] For example, if the depth value of a fusion position is 530.03 mm, and the two closest low-frequency measurement values ​​are 528.9 mm (sequence number 4) and 534.2 mm (sequence number 5), after inserting it between the two closest low-frequency measurement values, the ranking number of the fusion position 530.03 mm can be obtained as 5.

[0093] The relative depth change between this ranking number (5) and the ranking number (6) of the low-frequency measurement value of 537.1 mm of the pixel point is 1.

[0094] If the depth ranking of a fused position changes significantly compared to before fusion (low-frequency measurement value), for example, the relative depth change determined by the ranking number is 1000, it means that the pixel point has swapped the front and back relationship with more pixels after fusion.

[0095] It should be noted that, similar to step 2, the pixels to be evaluated by the depth logic may come from all pixels, or may be taken only from the pixels whose high-frequency and low-frequency distance difference after fusion in S201 is greater than a set threshold.

[0096] S204: Determine the final fusion depth comprehensively according to the fusion distance difference and the relative depth change.

[0097] In this step, for the pixel points to be evaluated by the depth logic, the reasonable fusion position can be comprehensively judged and the final depth map can be given based on the distance difference calculated in S201 and the relative depth change of the pixel after fusion calculated in S203.

[0098] The depth measurement method provided by the present application is introduced below in conjunction with specific embodiments.

[0099] In the following description, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the present application.

[0100] The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0101] The depth measurement method provided in the embodiments of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0102] Take the terminal device as a mobile phone as an example. Figure 3 The block diagram shows a partial structure of a mobile phone provided in an embodiment of the present application. Figure 3The mobile phone includes: a radio frequency (RF) circuit 310, a memory 320, an input unit 330, a display unit 340, a sensor 350, an audio circuit 360, a wireless fidelity (Wi-Fi) module 370, a processor 380, and a power supply 390. Those skilled in the art will understand that Figure 3 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0103] Combine the following Figure 3 A detailed introduction to the various components of the mobile phone:

[0104] The RF circuit 310 can be used for receiving and sending signals during the process of sending and receiving information or making calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 380 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the RF circuit 310 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE)), email, Short Messaging Service (SMS), etc.

[0105] The memory 320 can be used to store software programs and modules. The processor 380 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 320. The memory 320 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 320 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0106] The input unit 330 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone 300. Specifically, the input unit 330 may include a touch panel 331 and other input devices 332. The touch panel 331, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 331 or near the touch panel 331 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 331 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 380, and can receive and execute the command sent by the processor 380. In addition, the touch panel 331 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 331, the input unit 330 may also include other input devices 332. Specifically, other input devices 332 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0107] The display unit 340 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 340 may include a display panel 341. Optionally, the display panel 341 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 331 may cover the display panel 341. When the touch panel 331 detects a touch operation on or near it, it is transmitted to the processor 380 to determine the type of touch event. Subsequently, the processor 380 provides a corresponding visual output on the display panel 341 according to the type of touch event. Although in Figure 3 In the embodiment, the touch panel 331 and the display panel 341 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 331 and the display panel 341 can be integrated to realize the input and output functions of the mobile phone.

[0108] The mobile phone 300 may also include at least one sensor 350, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 341 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 341 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0109] The audio circuit 360, the speaker 361, and the microphone 362 can provide an audio interface between the user and the mobile phone. The audio circuit 360 can transmit the received audio data to the speaker 361 after converting the electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 362 converts the collected sound signal into an electrical signal, which is received by the audio circuit 360 and converted into audio data, and then the audio data is output to the processor 380 for processing, and then sent to another mobile phone through the RF circuit 310, or the audio data is output to the memory 320 for further processing.

[0110] Wi-Fi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the Wi-Fi module 370, which provides users with wireless broadband Internet access. Figure 3 A Wi-Fi module 370 is shown, but it is understandable that it is not an essential component of the mobile phone 300 and can be omitted as required without changing the essence of the invention.

[0111] The processor 380 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 320, and calling data stored in the memory 320, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 380 may include one or more processing units; preferably, the processor 380 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 380.

[0112] The mobile phone 300 may also include a camera 390, which may be a TOF camera. Optionally, the position of the camera 390 on the mobile phone 300 may be front-mounted or rear-mounted, which is not limited in the embodiment of the present application.

[0113] Optionally, the mobile phone 300 may include a single camera, a dual camera, or a triple camera, etc., which is not limited in the embodiments of the present application.

[0114] For example, the mobile phone 300 may include three cameras, one of which is a main camera, one is a wide-angle camera, and one is a telephoto camera.

[0115] Optionally, when the mobile phone 300 includes multiple cameras, all of the multiple cameras may be front-mounted, or all of the cameras may be rear-mounted, or some may be front-mounted and others may be rear-mounted, and this embodiment of the present application is not limited to this.

[0116] Although not shown, the mobile phone 300 also includes a power source (such as a battery) for supplying power to various components. The power source can be logically connected to the processor 380 through a power management system, so that the power management system can manage charging, discharging, and power consumption.

[0117] In addition, although not shown, the mobile phone 300 may also include a Bluetooth module, etc., which will not be described in detail here.

[0118] Figure 4Schematic diagram of the software structure of the mobile phone 300 of the embodiment of the present application. Taking the Android system as an example, in some embodiments, the Android system is divided into four layers, namely, the application layer, the application framework layer (framework, FWK), the system layer and the hardware abstraction layer, and the layers communicate with each other through software interfaces.

[0119] like Figure 4 As shown, the application layer may include a series of application packages, and the application packages may include short message, calendar, camera, video, navigation, gallery, call and other applications.

[0120] The application framework layer provides an application programming interface (API) and a programming framework for the application programs of the application layer. The application framework layer may include some predefined functions, such as a function for receiving events sent by the application framework layer.

[0121] like Figure 4 As shown, the application framework layer may include a window manager, a resource manager, a notification manager, and the like.

[0122] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make the data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0123] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0124] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0125] The application framework layer may also include:

[0126] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying pictures.

[0127] The phone manager is used to provide communication functions of the mobile phone 300, such as management of call status (including answering, hanging up, etc.).

[0128] The system layer may include multiple functional modules, such as sensor service module, physical state recognition module, 3D graphics processing library (such as OpenGL ES), etc.

[0129] The sensor service module is used to monitor the sensor data uploaded by various sensors at the hardware layer and determine the physical state of the mobile phone 300;

[0130] Physical state recognition module, used to analyze and recognize user gestures, faces, etc.;

[0131] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0132] The system layer may also include:

[0133] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0134] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0135] The hardware abstraction layer is a layer between hardware and software. The hardware abstraction layer can include display drivers, camera drivers, sensor drivers, etc., which are used to drive related hardware of the hardware layer, such as display screens, cameras, sensors, etc.

[0136] The following embodiments can be implemented on a mobile phone 300 having the above hardware structure / software structure. The following embodiments will take the mobile phone 300 as an example to illustrate the depth measurement method provided in the embodiments of the present application.

[0137] Reference Figure 5 , shows a schematic flowchart of the depth measurement method provided by an embodiment of the present application. As an example but not a limitation, the method can be applied to the above-mentioned mobile phone 300, which is equipped with a TOF camera that uses a multi-frequency mode to measure depth. The method may specifically include the following steps:

[0138] S501, respectively collecting depth images of a current scene at at least two measurement frequencies, where the at least two measurement frequencies include at least a first frequency and a second frequency, the first frequency is greater than the second frequency, and the depth image includes a plurality of pixels;

[0139] In the embodiment of the present application, the first frequency may be a high frequency modulation frequency, and the second frequency may be a low frequency modulation frequency. In order to ensure that the measurement range under the low frequency modulation frequency is sufficient to cover all depths of the current scene, the specific values ​​of the high and low frequency modulation frequencies may be determined according to the actual scene depth.

[0140] As a first example of this embodiment, the first frequency, that is, the high-frequency modulation frequency, can be configured as 60MHz, and the range in this measurement mode is about 2.5 meters, while the second frequency, that is, the low-frequency modulation frequency, can be configured as 20MHz, and the range in this measurement mode is about 7.5 meters. For most indoor scenes, the range of 7.5 meters is sufficient to cover all depths in the scene. If the scene depth is too large, a lower modulation frequency can be used as the low-frequency frequency according to actual needs.

[0141] In an embodiment of the present application, the TOF camera of the mobile phone can be controlled to collect a depth image of the current scene at two measurement frequencies of 20 MHz and 60 MHz respectively.

[0142] S502, determining multiple candidate fusion depths for each pixel, and determining a relative depth of each pixel;

[0143] For high-frequency modulation frequencies, due to their short measurement range, when the scene depth exceeds their range, the obtained high-frequency measurement values ​​may not be the actual measurement values ​​of the scene, but may be folded a certain number of times.

[0144] Therefore, for each pixel in the depth image, its high-frequency measurement value can be taken, and combined with the specific range of the high-frequency modulation frequency, multiple possible candidate fusion depths can be calculated.

[0145] In the embodiment of the present application, the first depth value of each pixel at the high-frequency modulation frequency can be obtained respectively, and then a plurality of candidate fusion depths of each pixel can be calculated according to the first depth value.

[0146] In a specific implementation, the measurement range corresponding to the high-frequency modulation frequency can be first determined, and then the sum of the first depth value of each pixel and multiple non-negative integer multiples of the measurement range corresponding to the high-frequency modulation frequency can be calculated respectively, and used as multiple candidate fusion depths for each pixel.

[0147] like Figure 6 FIG. 1 is a schematic diagram of a candidate fusion depth according to an embodiment of the present application. Figure 6For a high-frequency modulation frequency of 60 MHz, the high-frequency range is about 2.5 meters. For the first depth value of a certain pixel point measured, the sum of the first depth value and 0 times, 1 times, or 2 times the high-frequency range can be calculated respectively, and used as the three candidate fusion depths of the pixel point, that is, Figure 6 Candidate fusion position 0, candidate fusion position 1 and candidate fusion position 2 in .

[0148] In the embodiment of the present application, the relative depth of each pixel can be determined according to the measurement value corresponding to the low-frequency modulation frequency.

[0149] In a specific implementation, the second depth value of each pixel at a low-frequency modulation frequency can be obtained respectively, and then each pixel can be sorted in ascending order of the second depth value, and the initial sorting number of each pixel after sorting can be determined, so that the initial sorting number can be used as the relative depth of the corresponding pixel.

[0150] As shown in Table 1, it is an example table of relative depths of a pixel point in an embodiment of the present application. For each pixel point, the low-frequency measurement value of the pixel point, that is, the second depth value, can be used as a reference to sort the low-frequency measurement values ​​from small to large, so as to obtain the relative depth shown in Table 1.

[0151] Table 1:

[0152]

[0153] For example, if the depth value of a certain pixel measured at a low-frequency modulation frequency is 537.1 mm, then according to Table 1, its relative depth is 6.

[0154] S503, identifying a fusion depth of a target pixel point according to multiple candidate fusion depths and relative depths of the target pixel point, where the target pixel point is any one of the multiple pixel points;

[0155] In the embodiment of the present application, the relative depth of each pixel can be combined to identify the fusion depth closest to the true depth value of the pixel from multiple candidate fusion depths of the pixel. In this embodiment, the folded depth value can be restored by measuring the depth value at a high frequency (60MHz) as the final fusion depth.

[0156] In a specific implementation, for each candidate fusion depth of each pixel, the final fusion depth of the pixel can be determined according to the change of its position in the relative depth sequence table.

[0157] That is, the relative position change between each candidate fusion depth of the target pixel and the relative depth of the target pixel can be determined respectively, and then the candidate fusion depth corresponding to the minimum value of the relative position change is identified as the fusion depth of the target pixel. The change represents the change in the relative depth between pixels corresponding to the fusion position.

[0158] In a specific implementation, each candidate fusion depth of the target pixel point can be inserted into a sequence sorted according to the second depth value, and then the difference (absolute value) between the sorting number of each candidate fusion depth in the sequence and the initial sorting number of the target pixel point is calculated, and the difference in the sequence number value is used as the relative position change.

[0159] For example, if the three candidate fusion depths of a pixel are T0, T1 and T2, they can be inserted into the relative depth sequence shown in Table 1 according to the numerical values ​​of T0, T1 and T2, and then the changes between T0, T1 and T2 and the initial sorting number of the pixel can be judged one by one. The candidate fusion depth corresponding to the minimum difference between the serial number values ​​can be identified as the final fusion depth of the pixel.

[0160] Assume that the serial number of a pixel point in the relative depth sequence shown in Table 1 is 6. If the candidate fusion depth T0 is inserted into the sequence and its value is between sequence 3 and sequence 4, the serial number of the candidate fusion depth T0 can be determined as 4; if the candidate fusion depth T1 is inserted into the sequence and its value is between sequence 15 and sequence 16, the serial number of the candidate fusion depth T1 can be determined as 16; similarly, if the candidate fusion depth T2 is inserted into the sequence and its value is between sequence 37 and sequence 38, the serial number of the candidate fusion depth T2 can be determined as 38; then calculate the difference between each candidate fusion depth and the serial number value of the pixel point in the relative depth sequence one by one, and it can be obtained that the difference in the serial number value of T0 is 2 (6-4), the difference in the serial number value of T1 is 10 (16-6), and the difference in the serial number value of T2 is 32 (38-6). Since the difference in the serial number value of T0 is the smallest, T0 can be used as the fusion depth of the pixel point.

[0161] S504: Generate a target depth image of the current scene based on the fused depths of multiple target pixels.

[0162] After the final fusion depth of each pixel is determined according to the aforementioned steps, a target depth image of the current scene may be output based on the fusion depth of all the pixels.

[0163] In an embodiment of the present application, by respectively collecting depth images of the current scene at at least two measurement frequencies, and after determining multiple candidate fusion depths for each pixel and the relative depth of each pixel, the final fusion depth of the pixel can be determined based on the multiple candidate fusion depths and relative depths of each pixel, and then the target depth image of the current scene can be generated. This embodiment obtains the folding number of high-frequency measurement values ​​through the relative depth information provided by the low-frequency measurement values ​​of the pixels, and considers the changes in the front-to-back relationship between the pixels in the fusion calculation, which can effectively protect the original depth logic of the scene, and can provide a fusion result that best conforms to the original depth logic of the scene, thereby improving the accuracy of the depth measurement.

[0164] Reference Figure 7 , shows a schematic step flow chart of a depth measurement method provided by another embodiment of the present application, and the method may specifically include the following steps:

[0165] S701, respectively collecting depth images of a current scene at at least two measurement frequencies, where the at least two measurement frequencies include at least a first frequency and a second frequency, the first frequency is greater than the second frequency, and the depth image includes a plurality of pixels;

[0166] It should be noted that this method can be applied to terminal devices equipped with a TOF camera with multi-frequency modes, such as mobile phones, tablet computers, cameras, camcorders, etc. This embodiment does not limit the specific type of terminal devices.

[0167] In an embodiment of the present application, the first frequency may be a 60 MHz frequency of a TOF camera, which corresponds to a range of approximately 2.5 meters, and the second frequency may be 20 MHz, which corresponds to a range of approximately 7.5 meters.

[0168] In an embodiment of the present application, the TOF camera can be controlled to collect a depth image of the current scene at two measurement frequencies of 20 MHz and 60 MHz respectively.

[0169] S702, determining multiple candidate fusion depths for each pixel;

[0170] In an embodiment of the present application, the folded depth value can be restored by using the depth value measured by the high-frequency modulation frequency, and the weighted average value of the depth value measured by the low-frequency modulation frequency can be used as the final fusion depth.

[0171] Therefore, for each pixel in the depth image, its high-frequency measurement value can be taken, and combined with the specific range of the high-frequency modulation frequency, multiple possible candidate fusion depths can be calculated.

[0172] like Figure 8 FIG. 1 is a schematic diagram of candidate fusion depths according to another embodiment of the present application. Figure 8 In the above example, for each pixel point, its high-frequency measurement value can be taken as the candidate fusion depth corresponding to folding 0 times (candidate fusion position 0); then, its high-frequency measurement value plus 1 high-frequency range is taken as the candidate fusion depth corresponding to folding 1 time (candidate fusion position 1); similarly, its high-frequency measurement value plus 2 high-frequency ranges are taken as the candidate fusion depth corresponding to folding 2 times (candidate fusion position 2).

[0173] S703, respectively obtaining a second depth value of each pixel at a second frequency;

[0174] The second depth value of each pixel at the second frequency is the depth value of each pixel in the depth image corresponding to the low-frequency modulation frequency, that is, Figure 8 Low frequency measurements in .

[0175] S704, calculating the distance difference between each candidate fusion depth of each pixel and the second depth value;

[0176] for Figure 8 The three candidate fusion depths in can be used to calculate the distance difference between them and the low-frequency measurement value. That is, for candidate fusion position 0, the distance difference between it and the low-frequency measurement value (distance difference 0) is calculated; for candidate fusion position 1 and candidate fusion position 2, the distance difference between them and the low-frequency measurement value, i.e., distance difference 1 and distance difference 2, can also be calculated in the above manner.

[0177] The distance difference between each candidate fusion depth and its low-frequency measurement value can be used to determine the relative depth of the pixel.

[0178] S705, extracting the minimum value of the distance difference as the minimum fusion distance of the corresponding pixel point, and determining the relative depth of each pixel point according to the minimum fusion distance;

[0179] In the embodiment of the present application, for each pixel, the smallest difference between the distance differences between the multiple candidate fusion depths and the low-frequency measurement value of the pixel can be found as the minimum fusion distance. Then, all pixels are divided into reference pixels and non-reference pixels according to the minimum fusion distance.

[0180] In a specific implementation, a threshold may be set, and pixels whose minimum fusion distance is less than the preset threshold may be identified as reference pixels, while those reference points whose minimum fusion distance is greater than or equal to the threshold may be identified as non-reference pixels (other points).

[0181] For the reference pixel point, its low-frequency measurement value, that is, the second depth value, can be taken, and then all the reference pixels are sorted in ascending order of the second depth value, and the initial sorting number of each reference pixel point after sorting is determined, so that the initial sorting number can be used as the relative depth of the reference pixel point.

[0182] For non-reference pixels, the second depth value of the non-reference pixel can be inserted into the sorting sequence of the reference pixels. By determining the sorting number of the second depth value of the non-reference pixel in the sorting sequence of the reference pixels, the sorting number of the non-reference pixel can be used as the relative depth of the non-reference pixel.

[0183] After the relative depth of each pixel is determined, the final fusion depth of each pixel can be given according to the change of the relative depth.

[0184] S706. For any reference pixel, identify a candidate fusion depth corresponding to the minimum fusion distance of the reference pixel as the fusion depth of the reference pixel;

[0185] For a reference pixel, the candidate fusion depth corresponding to the minimum fusion distance of the pixel can be directly identified as the final fusion depth of the pixel.

[0186] For example, in Figure 8 If the minimum fusion distance is the distance difference 0, and the distance difference is less than the above threshold, the candidate fusion depth 0 corresponding to the distance difference 0 can be identified as the final fusion depth of the pixel.

[0187] S707. For any non-reference pixel, determine the relative position change between each candidate fusion depth of the non-reference pixel and the relative depth of the non-reference pixel, and identify the candidate fusion depth corresponding to the minimum value of the relative position change as the fusion depth of the non-reference pixel;

[0188] As for non-reference pixels, they can be used as points to be evaluated, and then the final fusion depth of each point to be evaluated is given according to the change in relative depth.

[0189] In an embodiment of the present application, the relative position change between each candidate fusion depth of a non-reference pixel and the relative depth of the pixel can be first determined, and then the candidate fusion depth corresponding to the minimum relative position change can be identified as the fusion depth of the pixel.

[0190] In a specific implementation, each candidate fusion depth of the point to be evaluated can be inserted into a reference pixel point sequence sorted according to the second depth value, and then the absolute value of the difference between the sorting number of each candidate fusion depth in the sequence and the relative depth of the point to be evaluated (the sorting number after the point to be evaluated is inserted into the reference pixel point sequence) is calculated, and the absolute value is used as the relative position change.

[0191] Then, the candidate fusion depth corresponding to the minimum relative position change is found as the final fusion depth of the point to be evaluated.

[0192] S708: Generate a target depth image of the current scene based on the fused depth of the reference pixel and the non-reference pixel.

[0193] After calculating the fusion depths of all reference pixels and non-reference pixels, a depth image of the current scene can be output according to the fusion depth of each pixel.

[0194] In an embodiment of the present application, the folding number of high-frequency measurement values ​​is obtained by using the relative depth information provided by the low-frequency measurement values ​​of the pixel points. It is possible to find areas with low fusion result accuracy on the fusion results of the prior art, calculate the depth values ​​of these areas according to the original depth logic of the scene, and improve the accuracy of the depth measurement.

[0195] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0196] Corresponding to the depth measurement method described in the above embodiment, Fig. 9 A structural block diagram of a depth measurement device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0197] Reference Fig. 9 The device can be applied to a terminal device, and specifically may include the following modules:

[0198] A depth image acquisition module 901 is used to respectively acquire depth images of a current scene at at least two measurement frequencies, where the at least two measurement frequencies include at least a first frequency and a second frequency, where the first frequency is greater than the second frequency, and the depth image includes a plurality of pixels;

[0199] A candidate fusion depth determination module 902 is used to determine multiple candidate fusion depths for each pixel; and

[0200] A relative depth determination module 903 is used to determine the relative depth of each pixel;

[0201] A fusion depth identification module 904 is used to identify the fusion depth of a target pixel point according to a plurality of candidate fusion depths and a relative depth of the target pixel point, where the target pixel point is any one of the plurality of pixel points;

[0202] The target depth image generating module 905 is configured to generate a target depth image of the current scene based on the fused depths of multiple target pixels.

[0203] In the embodiment of the present application, the candidate fusion depth determination module 902 may specifically include the following submodules:

[0204] A first depth value acquisition submodule, used to respectively acquire a first depth value of each pixel point at a first frequency;

[0205] The candidate fusion depth calculation submodule is used to calculate multiple candidate fusion depths of each pixel point according to the first depth value.

[0206] In the embodiment of the present application, the candidate fusion depth calculation submodule may specifically include the following units:

[0207] a measurement range determining unit, configured to determine a measurement range corresponding to the first frequency;

[0208] The candidate fusion depth calculation unit is used to respectively calculate the sum of the first depth value of each pixel point and multiple non-negative integer multiple values ​​of the measurement range corresponding to the first frequency, and use the sum of the first depth value and the multiple non-negative integer multiple values ​​as the multiple candidate fusion depths of each pixel point.

[0209] In the embodiment of the present application, the relative depth determination module 903 may specifically include the following submodules:

[0210] A second depth value acquisition submodule, used to respectively acquire a second depth value of each pixel point at a second frequency;

[0211] A pixel sorting submodule, used to sort each pixel in the order of the second depth value from small to large;

[0212] The first relative depth determination submodule is used to determine an initial sorting number of each pixel after sorting, and use the initial sorting number as the relative depth of the pixel.

[0213] In the embodiment of the present application, the fusion depth recognition module 904 may specifically include the following submodules:

[0214] A relative position change determination submodule is used to respectively determine the relative position change between each candidate fusion depth of the target pixel and the relative depth of the target pixel;

[0215] The fusion depth recognition submodule is used to recognize the candidate fusion depth corresponding to the minimum value of the relative position change as the fusion depth of the target pixel point.

[0216] In the embodiment of the present application, the relative position change amount determination submodule may specifically include the following units:

[0217] A candidate fused depth insertion unit, configured to insert each candidate fused depth of the target pixel into a sequence sorted according to the second depth value;

[0218] The relative position change calculation unit is used to calculate the difference between the sorting number of each candidate fusion depth in the sequence and the initial sorting number of the target pixel point, and use the difference in the number value as the relative position change.

[0219] In the embodiment of the present application, the relative depth determination module 903 may further include the following submodules:

[0220] A distance difference calculation submodule, used to calculate the distance difference between each candidate fusion depth of each pixel and the second depth value;

[0221] A minimum fusion distance extraction submodule is used to extract the minimum value of the distance difference as the minimum fusion distance of the corresponding pixel point;

[0222] The second relative depth determination submodule is used to determine the relative depth of each pixel according to the minimum fusion distance.

[0223] In the embodiment of the present application, the second relative depth determination submodule may specifically include the following units:

[0224] A reference pixel point identification unit, used to identify a pixel point whose minimum fusion distance is less than a preset threshold as a reference pixel point;

[0225] A reference pixel sorting unit, configured to sort the reference pixels in ascending order of the second depth values;

[0226] A reference pixel relative depth determination unit, used to determine an initial sorting number of each reference pixel after sorting, and use the initial sorting number as the relative depth of the reference pixel;

[0227] a non-reference pixel insertion unit, configured to insert, for any non-reference pixel, a second depth value of the non-reference pixel into a sorted sequence of the reference pixels;

[0228] The non-reference pixel relative depth determination unit is used to determine the sorting number of the second depth value of the non-reference pixel in the sorting sequence of the reference pixels, and use the sorting number as the relative depth of the non-reference pixel.

[0229] In the embodiment of the present application, the target pixel point may include the reference pixel point and the non-reference pixel point, and the fusion depth recognition module 904 may further include the following submodules:

[0230] A reference pixel fusion depth recognition submodule is used to recognize, for any reference pixel, a candidate fusion depth corresponding to the minimum fusion distance of the reference pixel as the fusion depth of the reference pixel;

[0231] The non-reference pixel fusion depth identification submodule is used to determine, for any non-reference pixel, the relative position change between each candidate fusion depth of the non-reference pixel and the relative depth of the non-reference pixel, and identify the candidate fusion depth corresponding to the minimum value of the relative position change as the fusion depth of the non-reference pixel.

[0232] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.

[0233] Reference Fig.10 , shows a schematic diagram of a terminal device according to an embodiment of the present application. Fig.10 As shown, the terminal device 1000 of this embodiment includes: a processor 1010, a memory 1020, and a computer program 1021 stored in the memory 1020 and executable on the processor 1010. When the processor 1010 executes the computer program 1021, the steps in each embodiment of the above-mentioned depth measurement method are implemented, for example Figure 5 Alternatively, when the processor 1010 executes the computer program 1021, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Fig. 9 Functions of modules 901 to 905 are shown.

[0234] Exemplarily, the computer program 1021 may be divided into one or more modules / units, and the one or more modules / units are stored in the memory 1020 and executed by the processor 1010 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments may be used to describe the execution process of the computer program 1021 in the terminal device 1000. For example, the computer program 1021 may be divided into a depth image acquisition module, a candidate fusion depth determination module, a relative depth determination module, a fusion depth recognition module, and a target depth image generation module, and the specific functions of each module are as follows:

[0235] A depth image acquisition module, used to respectively acquire depth images of the current scene at at least two measurement frequencies, wherein the at least two measurement frequencies include at least a first frequency and a second frequency, wherein the first frequency is greater than the second frequency, and the depth image includes a plurality of pixels;

[0236] a candidate fusion depth determination module, used to determine multiple candidate fusion depths for each pixel; and

[0237] A relative depth determination module, used to determine the relative depth of each pixel;

[0238] A fusion depth recognition module, used to identify the fusion depth of a target pixel point according to a plurality of candidate fusion depths and a relative depth of the target pixel point, wherein the target pixel point is any one of the plurality of pixel points;

[0239] The target depth image generation module is used to generate a target depth image of the current scene based on the fused depth of multiple target pixels.

[0240] The terminal device 1000 may be a mobile phone, tablet computer, camera, camcorder or other device equipped with a multi-frequency TOF camera. The terminal device 1000 may include, but is not limited to, a processor 1010 and a memory 1020. Those skilled in the art will understand that Fig.10 It is only an example of the terminal device 1000 and does not constitute a limitation of the terminal device 1000. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 1000 may also include input and output devices, network access devices, buses, etc.

[0241] The processor 1010 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0242] The memory 1020 may be an internal storage unit of the terminal device 1000, such as a hard disk or memory of the terminal device 1000. The memory 1020 may also be an external storage device of the terminal device 1000, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 1000. Further, the memory 1020 may also include both an internal storage unit of the terminal device 1000 and an external storage device. The memory 1020 is used to store the computer program 1021 and other programs and data required by the terminal device 1000. The memory 1020 may also be used to temporarily store data that has been output or is to be output.

[0243] The embodiment of the present application further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the depth measurement method of each of the aforementioned embodiments can be implemented.

[0244] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0245] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0246] In the embodiments provided in the present application, it should be understood that the disclosed depth measurement method, device and terminal device can be implemented in other ways. For example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0247] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0248] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0249] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the depth measurement device and the terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0250] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A depth measurement method, characterized in that: include: Respectively collecting depth images of the current scene at at least two measurement frequencies, the at least two measurement frequencies at least including a first frequency and a second frequency, the first frequency being greater than the second frequency, and the depth image including a plurality of pixel points; Obtaining a first depth value of each pixel at a first frequency respectively, calculating a plurality of candidate fusion depths of each pixel according to the first depth value, and determining a relative depth of each pixel; Determine the relative position change between each candidate fusion depth of the target pixel and the relative depth of the target pixel, and identify the candidate fusion depth corresponding to the minimum value of the relative position change as the fusion depth of the target pixel, where the target pixel is any one of the multiple pixels; Generate a target depth image of the current scene based on the fusion depth of multiple target pixels; Wherein, determining the relative depth of each pixel point includes: Obtain a second depth value of each pixel at a second frequency respectively, sort each pixel in ascending order of the second depth value, determine an initial sorting number of each pixel after sorting, and use the initial sorting number as the relative depth of the pixel.

2. The method according to claim 1, characterized in that The calculating, according to the first depth value, a plurality of candidate fusion depths of each pixel point includes: Determining a measurement range corresponding to the first frequency; The sum of the first depth value of each pixel and multiple non-negative integer multiple values ​​of the measurement range corresponding to the first frequency is calculated respectively, and the sum of the first depth value and the multiple non-negative integer multiple values ​​is used as multiple candidate fusion depths for each pixel.

3. The method according to claim 1, characterized in that: The respectively determining the relative position change between each candidate fusion depth of the target pixel and the relative depth of the target pixel includes: Inserting each candidate fusion depth of the target pixel into a sequence sorted according to the second depth value; The difference between the sorting number of each candidate fusion depth in the sequence and the initial sorting number of the target pixel is calculated, and the difference in the sorting number is used as the relative position change.

4. The method according to any one of claims 1 to 3, characterized in that: The determining the relative depth of each pixel also includes: Obtain a second depth value of each pixel at a second frequency respectively; Calculating a distance difference between each candidate fused depth of each pixel and the second depth value; Extracting the minimum value of the distance difference as the minimum fusion distance of the corresponding pixel points; The relative depth of each pixel is determined according to the minimum fusion distance.

5. The method according to claim 4, characterized in that The determining the relative depth of each pixel point according to the minimum fusion distance includes: Identify the pixel point whose minimum fusion distance is less than a preset threshold as a reference pixel point; Sort the reference pixels in ascending order of the second depth values; Determine an initial sorting number of each reference pixel after sorting, and use the initial sorting number as the relative depth of the reference pixel; For any non-reference pixel, insert the second depth value of the non-reference pixel into the sorted sequence of the reference pixel; Determine a sorting number of the second depth value of the non-reference pixel in a sorting sequence of the reference pixels, and use the sorting number as the relative depth of the non-reference pixel.

6. The method according to claim 5, characterized in that The target pixel includes the reference pixel and the non-reference pixel, and identifying the fusion depth of the target pixel according to a plurality of candidate fusion depths and the relative depth of the target pixel includes: For any reference pixel, identifying a candidate fusion depth corresponding to the minimum fusion distance of the reference pixel as the fusion depth of the reference pixel; For any non-reference pixel, the relative position change between each candidate fusion depth of the non-reference pixel and the relative depth of the non-reference pixel is determined respectively, and the candidate fusion depth corresponding to the minimum value of the relative position change is identified as the fusion depth of the non-reference pixel.

7. A depth measuring device, characterized in that: include: A depth image acquisition module, used to respectively acquire depth images of the current scene at at least two measurement frequencies, wherein the at least two measurement frequencies include at least a first frequency and a second frequency, wherein the first frequency is greater than the second frequency, and the depth image includes a plurality of pixels; a candidate fusion depth determination module, configured to respectively obtain a first depth value of each pixel at a first frequency, and calculate a plurality of candidate fusion depths of each pixel according to the first depth value; and A relative depth determination module, used to determine the relative depth of each pixel; a fusion depth recognition module, used to respectively determine a relative position change between each candidate fusion depth of a target pixel and a relative depth of the target pixel, and recognize a candidate fusion depth corresponding to a minimum value of the relative position change as a fusion depth of the target pixel, wherein the target pixel is any one of the multiple pixels; A target depth image generation module, used to generate a target depth image of the current scene based on the fused depth of multiple target pixels; Among them, the relative depth determination module is specifically used to: obtain the second depth value of each pixel point at the second frequency respectively, sort each pixel point in order from small to large according to the second depth value, determine the initial sorting number of each pixel point after sorting, and use the initial sorting number as the relative depth of the pixel point.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the depth measurement method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the depth measurement method according to any one of claims 1 to 6 is implemented.

Images