A high dynamic range depth ranging method, system, device, and medium

Abstract

The application relates to a high dynamic range depth ranging method, system, device and medium, the method comprising: performing pixel consistency correction on acquired original phase maps and original intensity maps to generate corrected phase maps and corrected intensity maps; generating two phase delay components according to the corrected phase maps, and obtaining two merged phase delay components through neighborhood window pixel merging; calculating a depth map DB through a depth solving function from the two merged phase delay components, and performing edge-guided clustering on the depth map DB to obtain a depth map DBedge; performing adaptive interpolation on the depth map DBedge to obtain an interpolated map; and generating a high dynamic range depth map based on the corrected phase maps, the corrected intensity maps and the interpolated map. The application optimizes the dynamic range of ranging through the means of neighborhood window pixel merging, reduces the missing of measurement frame rate and timeliness caused by multiple exposures, and improves the dynamic range of the measurement distance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a high dynamic range depth ranging method, system, device and medium. BACKGROUND

[0002] At present, the space perception of a depth camera is affected by the intensity of a reflected light signal, which is related to the reflectivity of a target object and a spatial distance. Effective depth information generally requires that the intensity of a feedback light signal cannot be too high to cause ranging errors, and cannot be too low to cause ranging information to be submerged by noise.

[0003] At least the following problems exist in the prior art: the most common method to increase the dynamic range of depth camera ranging is to perform multiple exposures, but multiple exposures will inevitably result in a lack of measurement frame rate and timeliness. SUMMARY

[0004] (I) Technical problems to be solved

[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present application provides a high dynamic range depth ranging method, system, device and medium, which solves the technical problems of ranging errors caused by overexposure of the feedback light signal of the existing depth camera or too narrow ranging dynamic range caused by too low.

[0006] (II) Technical solutions

[0007] In order to achieve the above-mentioned purposes, the main technical solutions adopted by the present application include:

[0008] In a first aspect, the present application provides a high dynamic range depth ranging method, comprising:

[0009] performing pixel consistency correction on an original phase map and an original intensity map obtained by a depth sensor to generate a corrected phase map and a corrected intensity map;

[0010] generating two phase delay components according to the corrected phase map, and performing neighborhood window pixel merging on the two phase delay components to obtain two merged phase delay components;

[0011] calculating a depth map DB from the two merged phase delay components through a depth solving function, and removing unreliable pixel depth values by performing edge-guided clustering on the depth map DB to obtain a depth map DBedge;

[0012] performing adaptive interpolation on the depth map DBedge to obtain an interpolated map;

[0013] generate a high dynamic range depth map based on the corrected phase map, the corrected intensity map, and the interpolated map.

[0014] Optionally, performing pixel consistency correction on the original phase map and the original intensity map obtained by the depth sensor to generate a corrected phase map and a corrected intensity map comprises:

[0015] obtaining a phase-related image collected by a depth sensor;

[0016] obtaining an original phase map and an original intensity map by analyzing the phase-related image;

[0017] performing pixel consistency correction on the original phase map and the original intensity map to obtain a corrected phase map and a corrected intensity map;

[0018] wherein the depth sensor comprises a sinusoidal wave modulation depth sensor and / or a square wave modulation depth sensor.

[0019] Optionally, generating two phase delay components from the corrected phase map, and performing neighborhood window pixel merging on the two phase delay components respectively to obtain two merged phase delay components comprises:

[0020] converting the phase of the corrected phase map into a phase corresponding to a measurement distance of one period at a current adjustment frequency;

[0021] for the sinusoidal wave modulation depth sensor, calculating two phase delay components of sine and cosine of the converted phase;

[0022] for the square wave modulation depth sensor, calculating two phase delay components of effective illumination response of the converted phase;

[0023] performing neighborhood window pixel merging on the two phase delay components obtained by the sinusoidal wave modulation depth sensor or the square wave modulation depth sensor respectively to obtain two merged phase delay components.

[0024] Optionally, calculating a depth map DB from the two merged phase delay components through a depth solving function, and removing unreliable pixel depth values by performing edge-guided clustering on the depth map DB to obtain a depth map DBedge comprises:

[0025] calculating the depth map DB from the two merged phase delay components through a depth solving function;

[0026] extracting an edge part of the depth map DB through a canny algorithm to obtain edge pixels of the depth map DB;

[0027] Removing edge pixels of the depth map DB and depth values unreliable pixels obtained by clustering adjacent pixels of the edge pixels from the depth map DB to obtain a depth map DBedge;

[0028] wherein the depth solving function is:

[0029]

[0030] wherein c is the speed of light, π is the ratio of a circle, F led is the modulation frequency of the emitted laser of the depth sensor, D ofset is other error offsets, X and Y are respectively the first merged phase delay component and the second merged phase delay component in the two merged phase delay components.

[0031] Optionally, performing adaptive interpolation on the depth map DBedge to obtain an interpolated map comprises:

[0032] Performing smoothing operation on the depth map DBedge to obtain a smoothed image of the depth map DBedge.

[0033] Performing bicubic interpolation on the smoothed image of the depth map DBedge to obtain an interpolated map.

[0034] Optionally, performing bicubic interpolation on the smoothed image of the depth map DBedge to obtain an interpolated map comprises:

[0035] Enlarging the smoothed image of the depth map DBedge by K times to obtain a depth image to be interpolated Depth.

[0036] Obtaining depth values of any pixel point (x, y) in the smoothed image of the depth map DBedge and 8 adjacent pixel points of the pixel point (x, y), and obtaining influence factors of the pixel point (x, y) and the 8 adjacent pixel points through an interpolation base function.

[0037] Based on the influence factors, obtaining a pixel value at (X, Y) corresponding to the pixel point (x, y) on the depth image to be interpolated Depth, that is, obtaining a pixel value of each pixel point of the depth image to be interpolated Depth.

[0038] According to the pixel value of each pixel point of the depth image to be interpolated Depth, obtaining a pseudo-high-resolution interpolated map.

[0039] Optionally, generating a high dynamic range depth map based on the corrected phase map, the corrected intensity map and the interpolated map comprises:

[0040] Determine the confidence of the pixel corresponding to the pixel coordinate position of the Dm map based on the corrected intensity map. When the intensity value is less than a set first threshold or greater than a set second threshold, it is considered that the data of the pixel is invalid, and it is directly set to zero.

[0041] Find the missing pixel points of the pseudo high-resolution interpolation map on the Dm map judged by confidence according to the smoothed image of the depth map DBedge.

[0042] Fill the found pixel points in the pseudo high-resolution interpolation map, and obtain a high dynamic range depth map through a filtering operation.

[0043] The Dm map is obtained by median filtering the corrected phase map.

[0044] In a second aspect, an embodiment of the present application provides a high dynamic range depth ranging system, comprising:

[0045] A pixel correction module is configured to perform pixel consistency correction on an original phase map and an original intensity map obtained by a depth sensor to generate a corrected phase map and a corrected intensity map.

[0046] A phase delay component neighborhood window pixel merging module is configured to generate two phase delay components based on the corrected phase map, and perform neighborhood window pixel merging on the two phase delay components to obtain two merged phase delay components.

[0047] A depth solving module is configured to calculate a depth map DB by a depth solving function based on the two merged phase delay components, remove unreliable pixels of depth values on the depth map DB and edge guided clusters to obtain a depth map DBedge.

[0048] An adaptive interpolation module is configured to perform adaptive interpolation on the depth map DBedge to obtain an interpolation map.

[0049] A high dynamic range depth map generation module is configured to generate a high dynamic range depth map based on the corrected phase map, the corrected intensity map and the interpolation map.

[0050] In a third aspect, an embodiment of the present application provides a high dynamic range depth ranging device, comprising: at least one database; and a memory in communication connection with the at least one database; wherein the memory stores instructions executable by the at least one database, and the instructions are executed by the at least one database to enable the at least one database to perform a high dynamic range depth ranging method as described above.

[0051] In a fourth aspect, an embodiment of the present application provides a computer readable medium having stored thereon computer executable instructions that, when executed by a processor, implement a high dynamic range depth ranging as described above.

[0052] (III) Advantages

[0053] The present application has the advantages that: the present application effectively enhances the detection capability in the case of weak exposure / low reflection by using the technical means of neighborhood window pixel merging, overcomes the technical problems of loss of measurement frame rate and timeliness caused by other technologies for increasing the measurement dynamic range, and thus achieves the technical effects of increasing the high dynamic range of depth map ranging without losing the measurement frame rate and timeliness. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 A flowchart of a high dynamic range depth ranging method provided by an embodiment of the present application;

[0055] Figure 2 A specific flowchart of step S1 of a high dynamic range depth ranging method provided by an embodiment of the present application;

[0056] Figure 3 A phase correlation diagram of a high dynamic range depth ranging method provided by an embodiment of the present application;

[0057] Figure 4 A corrected phase diagram of a high dynamic range depth ranging method provided by an embodiment of the present application;

[0058] Figure 5 A corrected intensity diagram of a high dynamic range depth ranging method provided by an embodiment of the present application;

[0059] Figure 6 A specific flowchart of step S2 of a high dynamic range depth ranging method provided by an embodiment of the present application;

[0060] Figure 7 A specific flowchart of step S3 of a high dynamic range depth ranging method provided by an embodiment of the present application;

[0061] Figure 8 A depth map DB of a high dynamic range depth ranging method provided by an embodiment of the present application;

[0062] Figure 9 A diagram operated by a canny algorithm of a high dynamic range depth ranging method provided by an embodiment of the present application;

[0063] Figure 10 A depth map DBedge of a high dynamic range depth ranging method provided by an embodiment of the present application;

[0064] Figure 11 A detailed flowchart illustrating step S4 of a high dynamic range depth ranging method provided in an embodiment of the present invention;

[0065] Figure 12 This is a schematic diagram illustrating the specific process of step S42 in a high dynamic range depth ranging method provided in an embodiment of the present invention.

[0066] Figure 13 The high dynamic range depth ranging method provided in this embodiment of the invention is a 3x3 region centered on an arbitrary pixel in the depth map DB;

[0067] Figure 14 A detailed flowchart of step S5 of a high dynamic range depth ranging method provided in an embodiment of the present invention;

[0068] Figure 15 A high dynamic range depth map is provided for a high dynamic range depth ranging method according to an embodiment of the present invention.

[0069] Figure 16 This is a schematic diagram of the overall process of a high dynamic range depth ranging method provided in an embodiment of the present invention. Detailed Implementation

[0070] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] like Figure 1 As shown in the embodiment of the present invention, a high dynamic range depth ranging method includes: First, performing pixel consistency correction on the original phase map and original intensity map acquired by the depth sensor to generate a corrected phase map and a corrected intensity map; Second, generating two phase delay components based on the corrected phase map, and merging the two phase delay components into two merged phase delay components by performing neighborhood window pixel merging on the two phase delay components respectively; Third, calculating a depth map DB by using a depth calculation function on the two merged phase delay components, and then removing unreliable pixels with depth values ​​by performing edge-guided clustering on the depth map DB to obtain a depth map DBedge, and then adaptively interpolating the depth map DBedge to obtain an interpolated map; Finally, generating a high dynamic range depth map based on the corrected phase map, the corrected intensity map, and the interpolated map.

[0072] This invention employs a neighboring window pixel merging technique, which effectively enhances the detection capability under weak exposure / low reflection conditions. It overcomes the technical problems caused by other techniques that increase the dynamic range of measurement, such as the loss of measurement frame rate and timeliness. Thus, it achieves the technical effect of improving the high dynamic range of depth map ranging without sacrificing the measurement frame rate and timeliness.

[0073] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0074] Specifically, the present invention provides a high dynamic range depth ranging method, comprising:

[0075] S1. Perform pixel consistency correction on the original phase map and original intensity map acquired by the depth sensor to generate a corrected phase map and a corrected intensity map.

[0076] like Figure 2 As shown, step S1 includes:

[0077] S11. Acquire the phase-correlation image collected by the depth sensor. Figure 3 The diagram shown is a phase correlation plot in RGB format.

[0078] S12. Obtain the original phase map and the original intensity map by analyzing the phase correlation image.

[0079] The depth sensors include sinusoidal modulation depth sensors and / or square wave modulation depth sensors. In the distance calculation process of ToF (ToF) principle cameras, different types of ToF cameras require different numbers and methods of phase correlation maps; conventional sinusoidal modulation ToF uses 4 DCS (Digital Cosmetic Sequences) images per frame for calculation; while square wave modulation ToF uses 9 images for calculation.

[0080] Furthermore, a sinusoidal-modulated depth sensor was used to acquire DCS phase-correlation images, and the following two types of images were calculated using four sets of DCS phase-correlation images:

[0081] The original phase diagram is: P = atan2(DCS1-DCS3, DCS2-DCS4);

[0082] The original intensity map is: A = sqrt[(DCS1 - DCS3)] 2 +(DCS2-DCS4) 2 ].

[0083] S13. Perform pixel consistency correction on the original phase map and the original intensity map to obtain the following:

[0084] like Figure 4 The corrected phase diagram shown is expressed as: D = calib(P).

[0085] As Figure 5 The corrected intensity map is shown as A = calib(A).

[0086] S2, generating two phase delay components according to the corrected phase map, and performing neighborhood window pixel merging on the two phase delay components respectively to obtain two merged phase delay components.

[0087] As Figure 6 As shown in step S2, it comprises:

[0088] S21, converting the phase of the corrected phase map into the phase corresponding to the measurement distance of one period at the current adjustment frequency.

[0089] S22a, for the sinusoidal wave modulation depth sensor, calculating the sine and cosine two phase delay components of the converted phase, and calculating the sine and cosine two phase delay components of the converted phase as follows:

[0090] The sine phase delay component is:

[0091] The cosine phase delay component is:

[0092] S22b, for the square wave modulation depth sensor, calculating the effective illumination response two phase delay components of the converted phase.

[0093] S23, performing neighborhood window pixel merging on the two phase delay components obtained by the sinusoidal wave modulation depth sensor or the square wave modulation depth sensor respectively to obtain two merged phase delay components.

[0094] Further, the sine phase component and the cosine two phase components phase delay components are respectively subjected to 4x4 pixel merging neighborhood window pixel merging to obtain the merged sine phase component and the merged cosine two merged phase components phase delay components:

[0095] The merged sine phase delay component is: X = bin44(x);

[0096] The merged cosine phase delay component is: Y = bin44(y).

[0097] S3, calculating the depth map DB through the depth solving function according to the two merged phase delay components, and removing the unreliable pixel of the depth value through the edge guided clustering of the depth map DB to obtain the depth map DBedge.

[0098] As Figure 7 As shown in step S3, it comprises:

[0099] S31, obtaining a depth map DB by a depth solving function from the two combined phase delay components, Figure 8 The depth map DB is shown.

[0100] The depth solving function is:

[0101]

[0102] In the formula, c is the speed of light, π is the circular ratio, F led is the modulation frequency of the depth sensor laser, D ofset is other error offset, X and Y are respectively the first combined phase delay component and the second combined phase delay component in the two combined phase delay components.

[0103] S32, extracting the edge part of the depth map DB by a canny algorithm to obtain the edge pixel of the depth map DB, Figure 9 The figure operated by the canny algorithm is shown.

[0104] S33, removing the edge pixel of the depth map DB and the depth value unreliable pixel obtained by clustering the adjacent pixels of the edge pixel on the depth map DB to obtain a depth map DBedge. Figure 10 The depth map DBedge is shown. Preferably, the depth value unreliable pixel obtained by clustering the adjacent pixels of the edge pixel and not satisfying the preset condition can be several independent pixels or a collection of multiple pixels, such as a block, a region, etc.

[0105] S4, obtaining an interpolation map by adaptively interpolating the depth map DBedge.

[0106] As shown in the figure, Figure 11 Step S4 includes:

[0107] S41, performing a smoothing operation on the depth map DBedge to obtain a smoothed image of the depth map DBedge.

[0108] S42, performing a bicubic interpolation on the smoothed image of the depth map DBedge to obtain an interpolation map.

[0109] Further, as shown in the figure, Figure 12 Step S42 includes:

[0110] S421, magnifying the smoothed image of the depth map DBedge by K times to obtain a pseudo-high-resolution depth image Depth.

[0111] S422, obtaining the depth values of any pixel point and the adjacent 8 pixel points of the pixel point in the smoothed image of the depth map DBedge, and obtaining the influence factors of the total 9 pixel points of the pixel point and the adjacent 8 pixel points by an interpolation base function.

[0112] S423. Based on the influence factor, obtain the pixel value of the corresponding pixel point on the depth image Depth to be interpolated, and thus obtain the pixel value of each pixel point on the depth image Depth to be interpolated.

[0113] S424. Based on the pixel value of each pixel in the depth image Depth to be interpolated, a pseudo-high resolution interpolated image is obtained.

[0114] In the specific sub-step of S42 above, the size of the depth map DB is l×n, and the size of the pseudo-high-resolution image Depth after being magnified K times is L×N, that is... The coordinate mapping formula between the coordinates (x, y) of any pixel in the depth map DB and the coordinates (X, Y) of any pixel in the pseudo-high-resolution image Depth is as follows:

[0115] x=(X-1)u+1

[0116] y = (Y-1)v + 1

[0117] In the formula, the horizontal interpolation distance is u. The vertical interpolation distance is v. At this point, x and y represent the horizontal and vertical coordinates of a single pixel. The depth values ​​of pixel (x, y) in the depth map DB, along with the depth values ​​of the 8 nearest pixels to (x, y), are used as parameters to calculate the depth value at (X, Y) in the pseudo-high-resolution depth image. The influence factor α of the 9 pixels in the 3×3 pixel block is obtained using the Bicubic basis function, thus yielding the pixel value at (X, Y) in the depth image Depth.

[0118] like Figure 13 As shown, pixel Q represents the low-resolution depth image at (X,Y) corresponding to the pre-high-resolution depth image D1. The position of point Q. The coordinates of point Q include a decimal part, so assume the coordinates of Q are (x+δ, y+η), where x and y represent the integer parts, and b... 2,2 The coordinates of the asterisk point in the grid are (x, y), where δ and η represent the decimal part, i.e., the distance from the asterisk point to point b in the graph. 2,2 The distance between the dots in the grid. The image shows the positions of 9 adjacent pixels, all expressed using b. ij This indicates that i = 1, 2, 3, and j = 1, 2, 3.

[0119] Based on the two-dimensional characteristics of pixels, calculations are performed separately for rows and columns according to the Bicubic basis function. The row formula is as follows:

[0120]

[0121] Column C can be obtained in the same way j , and the column formula is as follows:

[0122]

[0123] In the 3x3 window area, the influence factor of a certain pixel point is:

[0124] α i,j =C i (r2)C j (r1), i=1, 2, 3, j=1, 2, 3,

[0125] The initialization formula after bicubic interpolation is as follows:

[0126]

[0127] Where i and j are used to represent the position coordinates of the 9 pixel points in the 3x3 window area, which are positively correlated with the size of x and y, for example, i=1, j=1 represents the coordinate (x-1, y-1) position, and i=2, j=2 represents the coordinate (x, y) position.

[0128] S5, based on the corrected intensity map, the corrected depth map and the interpolation map, generate a high dynamic range depth map.

[0129] As shown in Figure 14 , step S5 includes:

[0130] S51, based on the corrected intensity map, determine the confidence of the pixel corresponding to the pixel coordinate position of the Dm map, when the intensity value is less than a set first threshold or greater than a set second threshold, it is considered that the data of the pixel is invalid, and is directly set to zero.

[0131] S51, find the missing pixel points of the pseudo-high-resolution interpolation map on the Dm map after confidence judgment according to the smoothed image of the depth map DBedge.

[0132] S52, fill the found pixel points on the pseudo-high-resolution interpolation map, and obtain a high dynamic range depth map through a filtering operation, Figure 15 showing a high dynamic range depth map.

[0133] Wherein, Dm map is a map obtained by median filtering the corrected phase map.

[0134] The initialization of the low-resolution depth map DB is realized by the above-mentioned bicubic interpolation method, and the pixel clustering of the pseudo high-resolution depth image Depth is also completed, and the pseudo high-resolution depth image is used as a guide matrix to fill part of the target points in the sparse matrix in the adaptive autoregressive model. Since it is a pseudo high-resolution depth image, there are problems such as no data on the image edge, and the present application fills the depth value of the corresponding coordinate on the Dm image on the Depth image according to the smooth image of DBedge. Of course, theoretically, the noise of the image edge information will be larger after the above steps are completed, and other filtering needs to be matched to realize the optimization of the depth map.

[0135] In addition, the embodiment of the present application provides a high dynamic range depth ranging system, comprising:

[0136] A pixel correction module is configured to perform pixel consistency correction on an original phase image and an original intensity image obtained by a depth sensor to generate a corrected phase image and a corrected intensity image.

[0137] A phase delay component neighborhood window pixel merging module is configured to generate two phase delay components according to the corrected phase image, and perform neighborhood window pixel merging on the two phase delay components to obtain two merged phase delay components.

[0138] A depth solving module is configured to calculate a depth map DB by a depth solving function using the two merged phase delay components, remove unreliable pixels of depth values obtained by edge pixels and edge guide clustering on the depth map DB, and obtain a depth map DBedge.

[0139] An adaptive interpolation module is configured to perform adaptive interpolation on the depth map DBedge to obtain an interpolation map.

[0140] A high dynamic range depth map generation module is configured to generate a high dynamic range depth map based on the corrected phase image, the corrected intensity image, and the interpolation map.

[0141] Meanwhile, the embodiment of the present application also provides a high dynamic range depth ranging device, comprising: at least one database; and a memory in communication connection with the at least one database; wherein the memory stores instructions executable by the at least one database, and the instructions are executed by the at least one database to enable the at least one database to perform the high dynamic range depth ranging method as described above.

[0142] In addition, the embodiment of the present application also provides a computer readable medium having computer executable instructions stored thereon, and the executable instructions are executed by a processor to implement the high dynamic range depth ranging as described above.

[0143] It is worth mentioning that the depth sensor described above includes one or more of a structured light sensor, a coded light sensor, a stereo vision sensor, a ToF camera, and a lidar.

[0144] In summary, the present application provides a high dynamic range depth ranging method, system, device and medium, such as Figure 16 As shown, specifically includes: original phase and original intensity correction, phase delay component neighborhood window pixel merging, depth map calculation, adaptive interpolation step. The original phase and original intensity correction step is: according to the original phase and original intensity map obtained by the depth sensor, the pixel consistency correction is carried out to generate the corrected phase map and the corrected intensity map; the phase delay component neighborhood window pixel merging step is: using the corrected phase map to generate two phase delay components, respectively, the neighborhood window pixel merging is carried out on the two components, and two merged phase delay components are obtained; the depth map calculation step is: using two merged phase delay components to calculate the depth map DB through the depth solving function, and then removing the depth value unreliable pixel by edge guided clustering on the depth map DB to obtain the depth map DBedge; the adaptive interpolation step is: using DB map to guide the edge, and then carrying out adaptive interpolation, and finally obtaining the depth map through filtering. The present application scheme adopts the method of neighborhood window pixel merging to optimize the dynamic range of ranging, and the neighborhood window pixel merging is to add the charges sensed by the adjacent pixels together, and to read out in a pixel mode, which can effectively enhance the detection ability in the weak exposure / low reflection condition, and increase the ranging range.

[0145] Since the system / device described in the above embodiments of the present application is the system / device used for implementing the method of the above embodiments of the present application, the specific structure and deformation of the system / device can be understood by those skilled in the art based on the method described in the above embodiments of the present application, and thus will not be described here. Any system / device used in the method of the above embodiments of the present application belongs to the scope of the present application.

[0146] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0147] The present application is described in reference to the flowchart and / or block diagram of the method, apparatus (system) and computer program product according to an embodiment of present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as a combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0148] It should be noted that any references made in the claims to an appended figure are incidental to the specification and serve only to facilitate understanding of the claims. The word "comprising" does not exclude the presence of elements or steps other than those listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of hardware comprising several distinct elements, and by means of a programmed computer. In the claims, the term comprising does not exclude the presence of other elements or steps than those listed in a claim. The word "first", "second", "third" etc. does only serve the purpose of differentiation and is not an indication of any order. The terms "first", "second", "third", etc. should be interpreted in a non-limiting manner.

[0149] Furthermore, it is noted that the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art will recognize that the disclosed embodiments and examples are not the only way to implement the present application and that, in fact, numerous modifications and alterations to the disclosed embodiments and examples are both desirable and possible. It is understood that business arrangements, such as licensing negotiations, can vary from one implementation to another, and that many modifications are likely to be made by a licensor and a licensee in varying implementations. It is intended that each embodiment and example disclosed in the specification represent a combination of features that can be combined in any way and / or adaption of the application in different implementations. It is intended that each feature disclosed in the specification (and / or the claims) can also be replaced by alternative features that are functionally and / or structurally similar. It is intended that the scope of the present application should only be limited by the claims and their equivalents.

[0150] Although preferred embodiments of the application have been described, those skilled in the art will recognize that additional modifications and variations are possible in light of the above teachings. It is therefore intended that the claims cover all such modifications and changes as fall within the true scope of this application. Accordingly, while the application has been particularly shown and described with reference to specific embodiments, it will be apparent to those skilled in the art that various modifications and changes can be made to the application without departing from the spirit and scope of the application. It is intended that the scope of the application should only be limited by the claims and their equivalents.

[0151] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A high dynamic range depth ranging method, characterized in that, include: Pixel consistency correction is performed on the original phase map and original intensity map acquired by the depth sensor to generate a corrected phase map and a corrected intensity map, including: acquiring a phase-correlated image acquired by the depth sensor; obtaining the original phase map and original intensity map by parsing the phase-correlated image; performing pixel consistency correction on the original phase map and the original intensity map to obtain the corrected phase map and corrected intensity map; wherein, the depth sensor includes: a sinusoidal wave modulated depth sensor; Two phase delay components are generated based on the corrected phase map. The two phase delay components are then merged into two merged phase delay components by performing neighborhood window pixel merging. This includes: converting the phase of the corrected phase map into the phase of the measurement distance corresponding to the next cycle of the current adjustment frequency; calculating the sine and cosine phase delay components of the converted phase; and merging the two obtained phase delay components into two merged phase delay components by performing neighborhood window pixel merging. The depth map DB is obtained by calculating the two merged phase delay components using a depth calculation function. Then, edge-guided clustering is performed on the depth map DB to remove pixels with unreliable depth values, resulting in a depth map DBedge. This process includes: obtaining the depth map DB by calculating the two merged phase delay components using a depth calculation function; extracting the edge portion of the depth map DB using the Canny algorithm to obtain edge pixels of the depth map DB; and removing the edge pixels of the depth map DB and pixels with unreliable depth values ​​obtained by clustering adjacent pixels of the edge pixels from the depth map DB to obtain the depth map DBedge. The depth calculation function is: ； In the formula, c is the speed of light, π is pi, and F led It is the modulation frequency of the laser emitted by the depth sensor, D ofset Other error offsets, X and Y are the first and second combined phase delay components, respectively, of the two combined phase delay components; An interpolated image is obtained by adaptive interpolation of the depth map DBedge; A high dynamic range depth map is generated based on the corrected phase map, the corrected intensity map, and the interpolation map.

2. The high dynamic range depth ranging method as described in claim 1, characterized in that, The adaptive interpolation of the depth map DBedge to obtain the interpolated map includes: A smoothing operation is performed on the depth map DBedge to obtain a smoothed image of the depth map DBedge; Bicubic interpolation is performed on the smoothed image of the depth map DBedge to obtain the interpolated image.

3. The high dynamic range depth ranging method as described in claim 2, characterized in that, Perform bicubic interpolation on the smoothed image of the depth map DBedge to obtain the interpolated image, which includes: The smoothed image of the depth map DBedge is magnified by a factor of K to obtain the depth image Depth to be interpolated; Obtain any pixel in the smoothed image of the depth map DBedge. and pixels The depth values ​​of 8 adjacent pixels are used to obtain the pixel value through an interpolation basis function. The influence factor of a total of 9 pixels, including 8 adjacent pixels; Based on the influence factor, the corresponding pixel on the depth image to be interpolated (Depth) is obtained. of By finding the pixel value at a given location, the pixel value of each pixel in the depth image Depth to be interpolated can be obtained. Based on the pixel value of each pixel in the depth image to be interpolated (Depth), a pseudo-high resolution interpolated image is obtained.

4. The high dynamic range depth ranging method as described in claim 3, characterized in that, Based on the corrected phase map, the corrected intensity map, and the interpolation map, generating a high dynamic range depth map includes: Based on the corrected intensity map, the confidence level of the pixel at the corresponding pixel coordinate position in the Dm map is determined. When the intensity value is less than the set first threshold or greater than the set second threshold, the data of the pixel is considered invalid and is directly set to zero. Based on the smoothed image of the depth map DBedge, the missing pixels of the pseudo-high resolution interpolation map are found on the Dm map with confidence judgment; The found pixels are filled into the pseudo-high resolution interpolated map, and a high dynamic range depth map is obtained by filtering. The Dm diagram is obtained by applying median filtering to the corrected phase diagram.

5. A high dynamic range depth ranging system, characterized in that, include: A pixel correction module is used to perform pixel consistency correction on the original phase image and original intensity image acquired by the depth sensor to generate a corrected phase image and a corrected intensity image. The module includes: acquiring a phase-correlated image acquired by the depth sensor; obtaining the original phase image and original intensity image by parsing the phase-correlated image; and performing pixel consistency correction on the original phase image and the original intensity image to obtain the corrected phase image and corrected intensity image. The depth sensor includes a sinusoidal wave modulated depth sensor. The phase delay component neighborhood window pixel merging module is used to generate two phase delay components based on the corrected phase map, and to merge the two phase delay components into two merged phase delay components by performing neighborhood window pixel merging, including: converting the phase of the corrected phase map into the phase of the measurement distance corresponding to the next cycle of the current adjustment frequency; calculating the sine and cosine phase delay components of the converted phase; and merging the two obtained phase delay components into two merged phase delay components by performing neighborhood window pixel merging. The depth calculation module is used to calculate a depth map DB by using a depth calculation function to calculate the two merged phase delay components, and to remove edge pixels and unreliable pixels with depth values ​​obtained by edge-guided clustering from the depth map DB to obtain a depth map DBedge. The module includes: obtaining the depth map DB by using a depth calculation function to calculate the two merged phase delay components; extracting the edge portion of the depth map DB using the Canny algorithm to obtain edge pixels of the depth map DB; and removing the edge pixels of the depth map DB and unreliable pixels with depth values ​​obtained by clustering adjacent pixels of the edge pixels from the depth map DB to obtain the depth map DBedge. The depth calculation function is: ； In the formula, c is the speed of light, π is pi, and F led It is the modulation frequency of the laser emitted by the depth sensor, D ofset Other error offsets, X and Y are the first and second combined phase delay components, respectively, of the two combined phase delay components; An adaptive interpolation module is used to adaptively interpolate the depth map DBedge to obtain an interpolated map. A high dynamic range depth map generation module is used to generate a high dynamic range depth map based on the corrected phase map, the corrected intensity map, and the interpolation map.

6. A high dynamic range depth ranging device, characterized in that, include: At least one database; And a memory that is communicatively connected to the at least one database; The memory stores instructions that can be executed by the at least one database, which are executed by the at least one database to enable the at least one database to perform a high dynamic range depth ranging method as described in any one of claims 1-4.

7. A computer-readable medium having computer-executable instructions stored thereon, characterized in that, When the executable instructions are executed by the processor, they implement a high dynamic range depth ranging method as described in any one of claims 1-4.

Images