A method, apparatus and electronic device for calculating the depth of speckle structure light

Abstract

The application provides a method and device for calculating the depth of speckle structure light and an electronic device. The method comprises: obtaining a reference speckle image at a known fixed depth; obtaining an actual speckle image of a measured object; obtaining an infrared image of the measured object as a guide image; calculating a corresponding cost value of each pixel based on the reference image and the target image; setting a sliding window to traverse the image using the guide image, and assigning a gray weight and a distance weight of each pixel in the sliding window; calculating an aggregated cost value of each pixel in the target image at each disparity corresponding to the sliding window according to the gray weight, the distance weight and the cost value; and obtaining a full-image disparity value of the target image according to the aggregated cost value. By increasing the infrared image as a guide, the aggregated cost is adaptively adjusted, the accuracy of corresponding point matching is improved, the error rate of edge region matching is reduced, and the recovery effect is optimized.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to a method, apparatus, and electronic device for calculating the depth of speckle structured light. Background Technology

[0002] With the development and maturation of 3D vision technology, various industries are placing higher demands on the automated acquisition of overall three-dimensional data information of object surfaces in real-world environments. 3D reconstruction methods based on speckle projection offer advantages such as high speed, single-frame reconstruction, strong anti-interference capabilities, and high data accuracy, showing promising application prospects in human body scanning, dental scanning, and rapid measurement. Summary of the Invention

[0003] This application aims to provide a method, apparatus, and electronic device for calculating the depth of speckle structured light. By adding an infrared image as a guide, the aggregation cost is adaptively adjusted, thereby improving the accuracy of corresponding point matching and optimizing the depth recovery effect of edge regions.

[0004] According to one aspect of this application, a method for calculating the depth of speckle structured light is proposed, comprising:

[0005] Obtain a reference speckle image at a known fixed depth;

[0006] A method for calculating the depth of speckle structured light, characterized by comprising:

[0007] Obtain a reference speckle image at a known fixed depth;

[0008] Acquire the actual speckle image of the object under test and the infrared image of the object under test;

[0009] Based on the reference speckle image, the actual speckle image, and the infrared image, obtain the full-image disparity value of the actual speckle image;

[0010] Based on the full-map disparity value, depth information is calculated to obtain the depth map of the object being measured.

[0011] According to some embodiments, the aforementioned method includes:

[0012] The step of obtaining the full-image disparity value of the actual speckle image based on the reference speckle image, the actual speckle image, and the infrared image includes:

[0013] The target image is obtained based on the actual speckle image of the object under test;

[0014] A reference image is obtained based on a reference speckle image at a known fixed depth.

[0015] Acquire an infrared image of the object under test as a guide map;

[0016] Based on the reference image, calculate the cost of each pixel in the target image under different parallaxes;

[0017] Set a sliding window to traverse the guide map and assign grayscale weights and distance weights to each pixel within the sliding window;

[0018] Based on the grayscale weight, the distance weight, and the cost value, calculate the aggregate cost value of each pixel in the target image within the corresponding sliding window;

[0019] Based on the aggregated generation value, obtain the full-map disparity value of the target map.

[0020] According to some embodiments, the aforementioned method includes:

[0021] The step of setting a sliding window to traverse the guide map and assign grayscale weights and distance weights to each pixel within the sliding window includes:

[0022] Based on the difference in grayscale between each pixel in the window and the center pixel, assign corresponding weights to each pixel in the window and establish a grayscale weight distribution function.

[0023] Based on the distance between each pixel in the window and the center pixel, assign corresponding weights to each pixel in the window and establish a distance weight distribution function.

[0024] According to some embodiments, the aforementioned method includes:

[0025] The step of calculating the aggregate cost value of each pixel in the target image within the corresponding sliding window based on the grayscale weight, the distance weight, and the cost value includes:

[0026] Adjust the grayscale weight and the distance weight according to a preset ratio to establish a joint weight function;

[0027] Based on the joint weighting function and the cost value corresponding to each pixel on the target image, the aggregate cost value under each disparity is calculated by looping within the disparity range.

[0028] According to some embodiments, the aforementioned method includes:

[0029] The process of obtaining a target image based on the actual speckle image of the object under test and obtaining a reference image based on a reference speckle image at a known fixed depth includes: performing binarization processing on the actual speckle image of the object under test and the reference speckle image respectively.

[0030] According to some embodiments, the aforementioned method includes:

[0031] The binarization process includes:

[0032] Calculate the difference in grayscale between each pixel and the center pixel within a circle of radius M.

[0033] If the difference in grayscale between N consecutive pixels on the circumference and the center pixel is greater than the threshold, then the pixel value of the center pixel is set to 1; otherwise, it is set to 0.

[0034] Where M and N are natural numbers.

[0035] According to some embodiments, the aforementioned method includes:

[0036] The step of calculating the cost value of each pixel in the target image under different disparities based on the reference image includes:

[0037] Looping within the disparity range relative to the reference image, calculate the cost value corresponding to each pixel in the target image under each disparity. The cost value is the absolute difference in grayscale between the pixel in the target image and the reference image under the current disparity.

[0038] According to some embodiments, the aforementioned method includes:

[0039] The full-map disparity value of the target image is obtained using the WTA algorithm.

[0040] According to another aspect of this application, an apparatus for calculating the depth of speckle structured light is provided, comprising:

[0041] The image acquisition module acquires a reference speckle image at a known fixed depth; acquires the actual speckle image of the object under test and the infrared image of the object under test;

[0042] The disparity acquisition module acquires the full-image disparity value of the speckle image based on the reference speckle image, the actual speckle image, and the infrared image.

[0043] The depth calculation module calculates depth information based on the disparity value of the entire image to obtain the depth map of the measured object.

[0044] According to another aspect of this application, a speckle structured light depth measurement device is provided, comprising:

[0045] A speckle projector used to project speckle patterns;

[0046] A supplemental light is used to provide an infrared light source;

[0047] The data acquisition module is used to acquire the actual speckle image of the object under test, the speckle image at a known fixed depth, and the infrared image of the object under test.

[0048] The apparatus for calculating the depth of speckle structured light, as described above, is used to perform depth calculations based on the actual speckle image of the object under test, the speckle image at a known fixed depth, and the infrared image of the object under test, in order to obtain a depth map of the object under test.

[0049] According to another aspect of this application, an electronic device is provided, comprising:

[0050] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described in any one of the above methods.

[0051] According to another aspect of this application, a computer program product is provided, comprising a computer program or instructions that, when executed by a processor, implement the method described in any one of the above methods.

[0052] According to the example embodiment of this application, by adding an infrared image as a guide, the aggregation cost is adaptively adjusted, thereby improving the accuracy of corresponding point matching and optimizing the recovery effect of edge region depth.

[0053] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0055] Figure 1 A schematic diagram of edge expansion according to an example embodiment of this application is shown.

[0056] Figure 2 A structural block diagram of a speckle structured light depth measurement device according to an example embodiment of this application is shown.

[0057] Figure 3 A schematic diagram of the structure of a speckle structured light depth measurement device according to an example embodiment of this application is shown.

[0058] Figure 4 A flowchart illustrating a method for calculating the depth of speckle structured light according to an example embodiment of this application is shown.

[0059] Figure 5 A schematic diagram of a value storage system according to an example embodiment of this application is shown.

[0060] Figure 6 A schematic diagram illustrating an image processing flow according to an example embodiment of this application is shown.

[0061] Figure 7A general flowchart of a speckle structured light depth calculation method according to an example embodiment of this application is shown.

[0062] Figure 8 A depth map of the scene under test is shown according to an example embodiment of this application.

[0063] Figure 9 A schematic diagram illustrating the principle of circumferential pixel selection according to an example embodiment of this application is shown.

[0064] Figure 10 A flowchart illustrating the binarization process according to an example embodiment of this application is shown.

[0065] Figure 11 A schematic diagram illustrating the binarization processing effect according to an example embodiment of this application is shown.

[0066] Figure 12 A block diagram of a speckle structured light depth calculation apparatus according to an example embodiment of this application is shown.

[0067] Figure 13 A block diagram of an electronic device according to an exemplary embodiment is shown. Detailed Implementation

[0068] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0069] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0070] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0071] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0072] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0073] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily essential for implementing this application, and therefore cannot be used to limit the scope of protection of this application.

[0074] Speckle 3D reconstruction technology requires pixel-by-pixel matching between captured speckle images and reference speckle images to obtain the disparity map between them. The core algorithm involved is image correlation technology based on local windows. Currently, commonly used methods include similarity measurement based on correlation functions or methods based on AD, Census cost calculation, aggregation, and matching. However, the former has a large computational load and is very time-consuming, while the latter is too sensitive to different window sizes and the objects have discontinuous depth regions. Due to the limitations of the algorithm's robustness, mismatches can occur, leading to edge dilation.

[0075] To solve problems such as disparity point mismatch and edge dilation (e.g.) Figure 1 Image (a) shows a scene of the tested object taken with a regular camera. Figure 1 As shown in (b) (a depth map with edge dilation problem), this application provides a speckle structured light depth calculation method. Compared with the prior art, this application improves the accuracy of corresponding point matching and optimizes the depth recovery effect of edge regions by adding an infrared image as a guide map and adaptively adjusting the aggregation cost.

[0076] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application.

[0077] Figure 2 A structural block diagram of a speckle structured light depth measurement device according to an example embodiment of this application is shown.

[0078] According to some embodiments, the device for speckle structured light depth measurement includes: a speckle projector 201, a fill light 202, a data acquisition module 203, and a speckle structured light depth calculation device 204.

[0079] The speckle projector 201 is used to project a speckle pattern, such as an infrared speckle pattern, to obtain a speckle image.

[0080] The supplementary light 202 is used to provide an infrared light source to obtain infrared images.

[0081] The data acquisition module 203 is used to acquire target images, guide images, and reference images, and may include RGB and IR cameras, etc.

[0082] The data acquisition module can be an IR camera module or an IR+RGB camera module, or it can be a combination of an IR module and other camera modules depending on the application requirements. When only speckle and infrared images are needed, the data acquisition module is an IR camera module. When the actual application also requires RGB images, the data acquisition module can be an IR camera module + RGB camera module. Depending on the actual application requirements, the data acquisition module can be any combination of an IR module + other camera modules.

[0083] The device 204 for speckle structured light depth calculation is used to perform depth calculation based on a target map, a guide map, and a reference map to obtain a depth map of the object being measured.

[0084] The above devices are used to obtain a speckle pattern of the object under test, and an infrared image is also obtained as a guide image. Since the infrared image and the speckle pattern are from the same scene, they can be used as the basis for calculating the window weight for subsequent matching.

[0085] In infrared images, a corresponding weight is assigned to each pixel based on the difference in grayscale between the pixel within the set window and the center pixel, establishing a color weight distribution function. A corresponding weight is assigned based on the distance between each pixel within the window and the center pixel, establishing a Euclidean distance weight function. The two weights are balanced according to requirements, and the final aggregation cost is calculated.

[0086] This scheme improves the accuracy of corresponding point matching and optimizes the recovery effect of edge region depth by adding infrared images as a guide and adaptively adjusting the aggregation cost.

[0087] Figure 3 A schematic diagram of the structure of a speckle structured light depth measurement device according to an example embodiment of this application is shown.

[0088] According to some embodiments, a fill light is projected onto the object under test, and the IR camera module acquires an infrared image; a speckle projector is projected onto the object under test, and the IR camera module acquires a speckle image. The obtained speckle image (target image) and infrared image are compared with a pre-obtained reference image to perform depth calculation.

[0089] In some applications, such as face recognition, RGB images are required, so the acquisition module may include an RGB camera module.

[0090] Figure 4 A flowchart illustrating a method for calculating the depth of speckle structured light according to an example embodiment of this application is shown.

[0091] See Figure 4 In S401, the target image, guide image, and reference image are obtained.

[0092] The process of acquiring a target image, a guide image, and a reference image includes: obtaining a target image based on the actual speckle image of the object under test; acquiring an infrared image of the object under test as a guide image; and obtaining a reference image based on a speckle image at a known fixed depth.

[0093] According to some embodiments, an infrared image, a speckle pattern, and a reference image of the object under test are acquired. The speckle pattern of the object under test, also known as the target image, is acquired through a data acquisition module, such as an IR camera, under a speckle projector. The infrared image is acquired under a supplementary light. Simultaneously with acquiring the speckle pattern of the object under test, an infrared image is acquired as a guide image. Since the guide image and the speckle pattern represent the same scene, they can provide a basis for calculating window weights in subsequent matching.

[0094] The reference image is a planar speckle pattern at a known fixed depth, pre-calibrated by the equipment, which has been binarized.

[0095] According to some embodiments, the reference image and the target image are binarized respectively.

[0096] According to some embodiments, the binarization process may include: traversing the image, taking each pixel as the center and a circle of radius M as the center, and calculating the difference in grayscale between each pixel on the circle and the center pixel. If the difference in grayscale between N consecutive pixels on the circle and the center pixel is greater than a threshold, then the pixel value of the center pixel is set to 1; otherwise, it is set to 0; where M and N are natural numbers.

[0097] Binarization can greatly reduce the influence of background grayscale, preserve speckle distribution characteristics, and is more conducive to matching.

[0098] In S403, based on the reference image, the cost of each pixel in the target image under different parallaxes is calculated.

[0099] According to some embodiments, the cost value corresponding to the current pixel in the target image is calculated iteratively within the disparity range relative to the current pixel in the reference image, for each disparity. The entire image is traversed, all cost values ​​are calculated, and stored for later calculation. Within a pre-given disparity range, a memory space for a three-dimensional matrix is ​​allocated, with a size equal to the image size and a height equal to the disparity range. The absolute difference in grayscale between the target image and the reference image for the current disparity is calculated iteratively, and this value is stored at the corresponding index position in the three-dimensional matrix. Figure 5 The diagram shows a storage illustration of the cost value.

[0100] The cost value can be expressed as C(q) = I(q) - I′(q,d), where C(q) is the cost value of a pixel in the target image under the current disparity d, I and I′ are the target image and the reference image, respectively, I(q) is the gray value of the pixel in the target image, and I′(q,d) is the gray value of the pixel in the reference image with disparity d to point q.

[0101] In S405, a sliding window is set to traverse the guide map, and grayscale weights and distance weights are assigned to each pixel within the sliding window.

[0102] During data acquisition, for example, while acquiring an image of the object under test using an IR camera, a frame of infrared image is also acquired. The discontinuous structure of the object under test itself and the depth changes between it and the background result in many distinct edge contours and sharp corner structures in the scene. Figure 6 The diagram shows a partial region of the infrared image (a). In the image, select a region with discontinuous edges, such as the area marked by the red line. The grayscale distribution at this red line is as follows: Figure 6 As shown in (b).

[0103] According to some embodiments, a grayscale weight distribution function is established by assigning corresponding weights to each pixel in the window based on the difference in grayscale between each pixel in the window and the center pixel, and a distance weight distribution function is established by assigning corresponding weights to each pixel in the window based on the distance between each pixel in the window and the center pixel.

[0104] The grayscale weight distribution function and distance weight function above are both Gaussian functions. See the Gaussian function distribution diagram for details. Figure 6 As shown in (c). The grayscale weight distribution diagram established using the above method is shown in [reference]. Figure 6 As shown in (d). The distance weight distribution map established using the above method is shown in [reference]. Figure 6 As shown in (e).

[0105] According to some embodiments, the choice of calculation window size affects the final disparity calculation result and may produce incorrect 3D topography. Utilizing grayscale changes in the guide map as the response function for edge detection can provide correct guidance in areas prone to mismatches, correcting disparity calculation results and preventing edge bulging and sharp corner blunting. Therefore, it allows for the use of a larger window, achieving better results in uniform regions without causing bulging effects in areas of depth discontinuity.

[0106] In S407, the aggregate cost of each pixel in the target image within the corresponding sliding window is calculated based on the grayscale weight, distance weight, and cost value.

[0107] According to some embodiments, grayscale weights and distance weights are adjusted according to a preset ratio to establish a joint weight function; based on the joint weight function and the cost value corresponding to each pixel on the target image, the aggregate cost value within the window corresponding to each pixel of the target image is calculated.

[0108] According to some embodiments, the method for establishing the joint weight function can be as follows: using a guide map, setting a window of appropriate size, and establishing the joint weight function w(p,q) of color weight and Euclidean distance weight distribution function based on the following formula (1), the calculation formula is as follows:

[0109]

[0110] Where p is the pixel coordinate of the center point of the guide map, q is the pixel coordinate of a certain pixel within the window centered at point p, I(p) and I(q) are the gray values ​​corresponding to that pixel, α is the weighting factor, and σ is the weighting factor. d Let σ be the standard deviation of the distance. r denoted as the grayscale standard deviation.

[0111] Based on formula (1), a discrete data table of weight function is established. Weights are assigned to window pixels by looking up the table, which effectively reduces the computational load of the algorithm and reduces the time consumption.

[0112] The above algorithm considers both distance and grayscale weights simultaneously, adaptively adjusting the calculation of the aggregation cost. Figure 6 Taking the window region shown in (a) as an example, see the adjusted adaptive weight distribution diagram. Figure 6 As shown in (f), the larger the difference between the pixel and the center, the smaller the weight ratio, and the more it can play a guiding role.

[0113] Commonly used window cost aggregation methods, such as direct summation of costs within a window, are prone to producing mismatched point pairs. After calculating the cost of each pixel under different parallaxes, the corresponding weights are calculated based on the color gradient and distance changes within the window at each pixel in the infrared image. The overall flow of the speckle structured light depth calculation method using the above processing method is as follows: Figure 7 As shown.

[0114] The above calculation method provides interfaces for spatial and grayscale balance parameter tuning, allowing adjustment of the influence ratio of grayscale and distance according to different needs, and finally weighted calculation of the aggregate cost within the window. This aggregate cost is more capable of evaluating the similarity of speckle distribution within the window at pixels, thus improving the matching accuracy.

[0115] According to some embodiments, the in-window cost aggregation value of each pixel is calculated based on formula (2) to obtain the full image disparity.

[0116]

[0117] Where C(q) = I(q) - I′(q,d) is the cost value of the pixel under the current disparity d, I and I′ are the target image and the reference image respectively, w(p,q) is the joint weight function, and C f (p) represents the aggregated value.

[0118] By indexing the pixel coordinates, the image is traversed based on the row-column loop (row-first loop, column-by-column loop under each row). The merging weight of the selected size window at the current pixel is calculated based on formula (1). The window cost aggregation value is calculated according to formula (2) under the current disparity by looping within the disparity range.

[0119] In S409, the full-map disparity value of the target map is obtained based on the aggregated cost value.

[0120] According to some embodiments, the optimal disparity is determined based on the WTA algorithm, and the full-image disparity is finally obtained.

[0121] Disparity calculation determines the optimal disparity value for each pixel by aggregating costs, typically using a winner-take-all (WTA) algorithm. The WTA algorithm selects the disparity corresponding to the minimum cost as the optimal disparity.

[0122] In the previous step S407, when calculating the aggregate cost, several variables can be set to always store the current best cost and its corresponding disparity. After iterating through all disparities corresponding to each pixel, the best cost and disparity for that pixel can be obtained. The aggregate cost for that point is calculated iteratively within the disparity range, and this value is compared with the aggregate cost stored in the previous iteration of the variables. The minimum value is retained, and the corresponding disparity is saved. The disparity with the minimum aggregate cost is obtained at the end of the loop. After traversing the image, the total image disparity can be obtained.

[0123] In S411, depth information is calculated based on the full-map disparity value to obtain the depth map of the object being measured.

[0124] Based on the disparity value and the geometric relationship of the device, i.e., formula (3), the depth information is calculated, and finally the depth map of the measured scene is obtained.

[0125]

[0126] Depth is the depth, CD is the distance between the camera and the projector, L is the distance between the camera plane and the reference plane, f is the camera focal length, and dis is the parallax.

[0127] Figure 8 A depth map of the scene under test is shown according to an example embodiment of this application.

[0128] By utilizing grayscale variations in the guide map as the response function for edge detection, correct guidance is provided in areas prone to mismatches, thus correcting disparity calculation results and preventing edge inflation and sharp corner blunting. Furthermore, better results are obtained in uniform regions, avoiding inflation effects near areas of depth discontinuity. The depth map of the tested scene obtained using this method is shown below. Figure 8 As shown, and Figure 1 Compared to the depth map with edge inflation problem (b), the effect of removing edge inflation is obvious.

[0129] Figure 9 A schematic diagram illustrating the principle of circumferential pixel selection according to an example embodiment of this application is shown.

[0130] See Figure 9 According to some embodiments, there are 16 circular pixels with a radius of a set value, such as 3, centered on a certain pixel. Several (e.g., 16) circular pixels are selected around each pixel, and the difference in gray level between each surrounding pixel and the center pixel is calculated.

[0131] Figure 10 A flowchart illustrating the binarization process according to an example embodiment of this application is shown.

[0132] According to some embodiments, the specific process of binarization is as follows:

[0133] Traverse the image. Take a certain pixel as the center and form a circle of 16 pixels with a radius of a set value, such as 3. Select a number of (e.g., 16) circumferential pixels around each pixel. Calculate the difference between the gray level of each surrounding pixel and the center pixel. Set an appropriate threshold. If a number of consecutive pixels, such as 12 pixels, have a gray level difference greater than the threshold, set the value of that pixel to 1; otherwise, set it to 0.

[0134] Binarization can remove the influence of background grayscale while preserving speckle distribution characteristics, which is more conducive to matching. The effect of processing the target image using the above binarization method is illustrated in the following diagram. Figure 11 As shown.

[0135] It should be clearly understood that this application describes how specific examples are formed and used, but this application is not limited to any details of these examples. Rather, based on the teachings of the disclosure of this application, these principles can be applied to many other embodiments.

[0136] Those skilled in the art will understand that all or part of the steps of the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the program that performs the functions defined by the methods provided in this application can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk.

[0137] Furthermore, it should be noted that the above figures are merely illustrative representations of the processes included in the method according to exemplary embodiments of this application, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0138] Through the description of the exemplary embodiments, those skilled in the art will readily understand that the method for calculating speckle structured light depth according to the embodiments of this application has at least one or more of the following advantages.

[0139] According to the example embodiment, by adding an infrared image as a guide, the aggregation cost is adaptively adjusted, thereby improving the accuracy of corresponding point matching and optimizing the recovery effect of edge region depth.

[0140] The following describes an apparatus embodiment of this application, which can be used to perform the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0141] Figure 12 A block diagram of an apparatus for calculating the depth of a speckle structured light according to an exemplary embodiment is shown. Figure 12 The apparatus shown can perform the aforementioned method for calculating the depth of speckle structured light according to the embodiments of this application.

[0142] like Figure 12 As shown, the apparatus for calculating the depth of speckle structured light may include: an image acquisition module 1210, a parallax acquisition module 1220, and a depth calculation module 1230.

[0143] See Figure 12 Referring to the preceding description, the image acquisition module 1210 acquires a reference speckle image at a known fixed depth; acquires the actual speckle image of the object under test and the infrared image of the object under test;

[0144] The parallax acquisition module 1220 acquires the full-image parallax value of the actual speckle image based on the reference speckle image, the actual speckle image, and the supplementary infrared image.

[0145] The depth calculation module 1230 calculates depth information based on the full image disparity value to obtain a depth map of the measured object, including post-processing operations to optimize the depth image.

[0146] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.

[0147] Figure 13 A block diagram of an electronic device according to an exemplary embodiment is shown.

[0148] The following reference Figure 13 To describe an electronic device 200 according to this embodiment of the present application. Figure 13 The electronic device 200 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0149] like Figure 13 As shown, the electronic device 200 is presented in the form of a general-purpose computing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including storage unit 220 and processing unit 210), a display unit 240, etc.

[0150] The storage unit stores program code, which can be executed by the processing unit 210, causing the processing unit 210 to perform the methods described in this specification according to various exemplary embodiments of this application.

[0151] Storage unit 220 may include readable media in the form of volatile storage units, such as random access memory (RAM) 2201 and / or cache memory 2202, and may further include read-only memory (ROM) 2203.

[0152] Storage unit 220 may also include a program / utility 2204 having a set (at least one) program module 2205, such program module 2205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0153] Bus 230 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0154] Electronic device 200 can also communicate with one or more external devices 300 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 200, and / or with any device that enables electronic device 200 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 250. Furthermore, electronic device 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 260. Network adapter 260 can communicate with other modules of electronic device 200 via bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0155] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. The technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the methods described above according to the embodiments of this application.

[0156] Software products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0157] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0158] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0159] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0160] Exemplary embodiments of this application have been specifically shown and described above. It should be understood that this application is not limited to the detailed structures, arrangements, or implementation methods described herein; rather, this application is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A method for calculating the depth of speckle structured light, characterized in that, include: Obtain a reference speckle image at a known fixed depth; Acquire the actual speckle image of the object under test and the infrared image of the object under test; Based on the reference speckle image, the actual speckle image, and the infrared image, obtain the full-image disparity value of the actual speckle image; Based on the full-map disparity value, depth information is calculated to obtain the depth map of the object being measured; The step of obtaining the full-image disparity value of the actual speckle image based on the reference speckle image, the actual speckle image, and the infrared image includes: The target image is obtained based on the actual speckle image of the object under test; A reference image is obtained based on a reference speckle image at a known fixed depth. Acquire an infrared image of the object under test as a guide map; Based on the reference image, calculate the cost of each pixel in the target image under different parallaxes; Set up a sliding window to traverse the guide map, and assign grayscale weights and distance weights to each pixel in the sliding window, including: assigning corresponding weights to each pixel in the window according to the difference in grayscale between each pixel in the window and the center pixel, and establishing a grayscale weight distribution function; and assigning corresponding weights to each pixel in the window according to the distance between each pixel in the window and the center pixel, and establishing a distance weight distribution function. Based on the grayscale weight, the distance weight, and the cost value, calculate the aggregate cost value of each pixel in the target image within the corresponding sliding window, including: adjusting the grayscale weight and the distance weight according to a preset ratio, establishing a joint weight function, and calculating the aggregate cost value under each disparity by looping within the disparity range based on the joint weight function and the cost value corresponding to each pixel in the target image. Based on the aggregated generation value, obtain the full-map disparity value of the target map.

2. The method according to claim 1, characterized in that, The process of obtaining a target image based on the actual speckle image of the object under test and a reference image based on a reference speckle image at a known fixed depth includes: performing binarization processing on the actual speckle image of the object under test and the reference speckle image respectively.

3. The method according to claim 2, characterized in that, The binarization process includes: Calculate the difference in grayscale between each pixel and the center pixel within a circle of radius M. If the difference in grayscale between N consecutive pixels on the circumference and the center pixel is greater than the threshold, then the pixel value of the center pixel is set to 1; otherwise, it is set to 0. Where M and N are natural numbers.

4. The method according to claim 1, characterized in that, The step of calculating the cost value of each pixel in the target image under different disparities based on the reference image includes: Looping within the disparity range relative to the reference image, calculate the cost value corresponding to each pixel in the target image under each disparity. The cost value is the absolute difference in grayscale between the pixel in the target image and the reference image under the current disparity.

5. The method according to claim 1, characterized in that, The step of obtaining the full-map disparity value of the target map based on the aggregated generation value includes: The disparity corresponding to the minimum aggregation cost is obtained according to the WTA algorithm, and the full-map disparity value of the target map is obtained.

6. A device for calculating the depth of speckle structured light, characterized in that, include: The image acquisition module acquires a reference speckle image at a known fixed depth; Acquire the actual speckle image of the object under test and the infrared image of the object under test; The disparity acquisition module acquires the full-image disparity value of the speckle image based on the reference speckle image, the actual speckle image, and the infrared image. The depth calculation module calculates depth information based on the full-map disparity value to obtain the depth map of the object being measured. The step of obtaining the full-image disparity value of the actual speckle image based on the reference speckle image, the actual speckle image, and the infrared image includes: The target image is obtained based on the actual speckle image of the object under test; A reference image is obtained based on a reference speckle image at a known fixed depth. Acquire an infrared image of the object under test as a guide map; Based on the reference image, calculate the cost of each pixel in the target image under different parallaxes; Set up a sliding window to traverse the guide map, and assign grayscale weights and distance weights to each pixel in the sliding window, including: assigning corresponding weights to each pixel in the window according to the difference in grayscale between each pixel in the window and the center pixel, and establishing a grayscale weight distribution function; and assigning corresponding weights to each pixel in the window according to the distance between each pixel in the window and the center pixel, and establishing a distance weight distribution function. Based on the grayscale weight, the distance weight, and the cost value, calculate the aggregate cost value of each pixel in the target image within the corresponding sliding window, including: adjusting the grayscale weight and the distance weight according to a preset ratio, establishing a joint weight function, and calculating the aggregate cost value under each disparity by looping within the disparity range based on the joint weight function and the cost value corresponding to each pixel in the target image. Based on the aggregated generation value, obtain the full-map disparity value of the target map.

7. A speckle structured light depth measurement device, characterized in that, include: A speckle projector used to project speckle patterns; A supplemental light is used to provide an infrared light source; The data acquisition module is used to acquire the actual speckle image of the object under test, the reference speckle image at a known fixed depth, and the infrared image of the object under test. The apparatus for calculating the depth of speckle structured light as described in claim 6 is used to perform depth calculation based on the actual speckle image of the object under test, the reference speckle image at a known fixed depth, and the infrared image of the object under test, so as to obtain a depth map of the object under test.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-5.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the method as described in any one of claims 1-5.

Images