Method and apparatus for detecting edges in active stereoscopic images

Abstract

A method and apparatus for generating a three-dimensional depth map is provided. The method includes the steps of: (i) illuminating a target by a pattern projector having a background intensity or by a combination of a flood projector and a pattern projector without a background intensity operating together; (ii) capturing at least one image including one or more objects present at the illuminated target; (iii) converting the at least one captured image into data; (iv) processing the data received from the conversion of the at least one captured image while filtering the projected pattern from the processed data; (v) detecting edges of at least one of the one or more objects present at the illuminated target; and (vi) generating a three-dimensional depth map including at least one of the one or more objects for which edges have been determined.

Description

Technical Field

[0001] This disclosure generally relates to methods using optical devices, and more specifically, to the processing of images of a scene illuminated by a pattern projector. Background Technology

[0002] A stereo camera device is a component consisting of two camera units assembled within a stereo module. Stereo vision (also known as "stereoscopic vision" or "3D imaging") is a technique used to determine the depth of an image using stereoscopic images. In other words, it is the depth impression perceived by a person with normal binocular vision when viewing a scene with both eyes; binocular vision results in two slightly different images of the scene being created in each eye due to the different positions of the eyes / camera.

[0003] Active stereo vision is a form of stereo vision that actively employs light-emitting devices such as laser diodes or devices configured to emit structured light (e.g., patterned light) to simplify stereo matching problems.

[0004] One of the problems associated with this type of technology is edge detection, a crucial factor in detecting the boundaries of objects included in images retrieved using a stereo camera setup. Edge boundary detection can improve the matching process between images retrieved from two cameras, as the matching window may be cropped by edge boundaries (pixels outside the edge boundaries may belong to objects / environment at different depths or parallaxes), and therefore will not be matched with pixels present within the matching window, assuming that pixels within the matching window have very similar depths or parallaxes. Furthermore, edges can be used in post-processing stages to improve the final 3D depth map obtained.

[0005] Several problems exist associated with this processing of edges in active stereo images. To name just a few, edge intensity decreases with increasing distance from the camera device; when using a device configured to emit structured light, most edges in the retrieved image belong to the projected pattern rather than the object's edge; and the pattern density is too low to specify object boundaries at pixel resolution. One option to address this problem is to use a projector with enhanced background light intensity to detect edges in the active image.

[0006] To address the edge detection problem, numerous solutions have been proposed in this field. Some of these solutions are described below.

[0007] US20120114225 discloses an image processing apparatus that detects occlusion boundaries between objects within a depth image by applying an edge detection algorithm to the depth image. The image processing apparatus classifies occlusion boundaries into foreground region boundaries and background region boundaries using the depth gradient vector direction of the occlusion boundaries, and extracts the occlusion regions of the input depth image using the foreground region boundaries.

[0008] US20120269458 describes a method for interpolating and filtering a reduced-resolution depth image to recover a high-resolution depth image using edge information, wherein each depth image comprises an array of pixels at various locations, and wherein each pixel has a depth. First, the reduced depth image is upsampled by interpolating the missing locations by repeating the nearest neighbor depth values. Next, a moving window is applied to the pixels in the upsampled depth image. The window covers a set of pixels centered on each pixel. The pixels covered by the window are selected based on their relative offsets to depth edges, and only pixels on the same side of the depth edge of the center pixel are used in the filtering process.

[0009] However, the problem that needs to be solved is detecting edges included in the captured image of a scene illuminated by a pattern projector in an active stereoscopic implementation.

[0010] This invention proposes a solution for implementing a new edge detection mechanism under these circumstances. Summary of the Invention

[0011] The disclosure can be summarized by referring to the appended claims.

[0012] One object of this disclosure is to provide a method and apparatus capable of removing a projected pattern from an image being processed while preserving edge positions in the image being processed.

[0013] Another object of this disclosure is to provide a method and apparatus for detecting the edges of objects within an image including a background pattern.

[0014] Another object of this disclosure is to provide a method and apparatus for generating a three-dimensional depth map having high-quality edges of objects included in the image.

[0015] Another object of this disclosure is to provide a method and apparatus for eliminating data associated with a projection pattern included as part of a scene captured by a stereoscopic device.

[0016] Other objects of the present invention will become apparent from the following description.

[0017] According to a first embodiment of this disclosure, a method for generating a three-dimensional depth map is provided, comprising the following steps:

[0018] (i) Illuminate the target by a pattern projector with background intensity or by a combination of a floodlight projector and a pattern projector without background intensity operating together;

[0019] (ii) Capture at least one image, the at least one image comprising one or more objects present at the illuminated target;

[0020] (iii) Convert at least one captured image into data;

[0021] (iv) Process the data received from the conversion of the at least one captured image, and filter out the projection pattern from the processed data;

[0022] (v) Detecting the edge of at least one of one or more objects present at the illuminated target; and

[0023] (vi) Generate a three-dimensional depth map, which includes at least one object whose edges have been determined among the one or more objects.

[0024] In this document, the term "having background intensity" as used in connection with a projector throughout the specification and claims should be understood to refer to a projector capable of emitting background illumination with a non-zero intensity.

[0025] Throughout this specification and claims, the term "active stereoscopic image" should be understood to encompass both active stereoscopic images and / or structured light images.

[0026] According to this disclosure, after removing projected patterns (e.g., speckle patterns, stripe patterns, etc.) from an image captured by an image capturing device, edges are detected using background intensity. In other words, background intensity provides edges at pixel resolution.

[0027] According to another embodiment, the step of detecting the edge of at least one of one or more objects present at the illuminated target is performed by detecting non-patterned edges included in the at least one image.

[0028] According to another embodiment, step (iv) is performed by applying an algorithm configured to detect blobs included in the at least one image.

[0029] Throughout this specification and claims, the term "trace" is used to refer to the connected components of pixels strongly illuminated by a pattern projector (as opposed to a weakly illuminated background). The shape of the trace is not limited to a circle, but can be any suitable shape. An appropriate algorithm should be selected based on the shape, and this algorithm will be implemented as described herein.

[0030] In yet another embodiment, step (iv) further includes filling the pixels of the projected pattern belonging to each pattern clump with pixels, and determining the minimum value of all pixels included in such clumps.

[0031] According to yet another embodiment, the provided method further includes the following steps for each pixel included in a particular clump:

[0032] (a) Collect data relating to all pixels included within a square window surrounding the pixel;

[0033] (b) Remove all data from the collected data relating to pixels whose grayscale value is higher than the minimum value of all pixels included in the particular patch;

[0034] (c) Apply a bilateral filter to the collected data that was not removed in step (b); and

[0035] (d) Perform the following steps on the entire captured image:

[0036] (d1) Apply an operator to the obtained result, wherein the operator is configured to remove the projection pattern from the captured image while preserving the edge positions in the captured image. Such an operator could be, for example, a suitable opening operator; and

[0037] (d2) Apply edge detection operators (e.g., Canny edge detector) to patternless data to detect various edges in the captured image.

[0038] According to yet another embodiment, the provided method further includes the step of determining key points (e.g., intersections of lines (e.g., straight lines) detected in a patterned background) in an actively illuminated image. Straight lines can be fitted to edge lines suitable for such a fit (e.g., edges that are nearly linear) to detect appropriate straight line edges on a straight line object. Additionally or alternatively, key points (also called points of interest) can be detected within an image having a specific defined pattern, for example, by making the straight lines intersect. An advantage of implementing the key point detection method is that it allows for timely frame registration. This problem arises when dealing with active stereo systems because the pattern moves with the projector, meaning that identifying key points in different frames is a considerable challenge. The key point detection method enables another embodiment of this disclosure, in which points of interest (e.g., edge points, key points) are detected using the background intensity in the active stereo image.

[0039] According to another aspect of this disclosure, an apparatus for generating a three-dimensional depth map is provided, wherein the apparatus includes:

[0040] At least one projector, which is a pattern projector with background intensity or a combination of a floodlight projector and a pattern projector without background intensity configured to operate together, is suitable for illuminating a target while projecting a pattern at the target, where one or more objects are present.

[0041] At least one image capture sensor is configured to capture at least one image, the at least one image comprising one or more objects present at the illuminated target; and

[0042] At least one processor is configured to operate on the at least one image captured by the at least one image capture sensor, and is configured to:

[0043] (a) Process at least one captured image and convert it into data;

[0044] (b) Filter out the projected pattern from the processed data;

[0045] (c) Detecting the edge of at least one of one or more objects present at the irradiated target; and

[0046] (d) Generate a three-dimensional depth map, which includes at least one object whose edges have been detected among the one or more objects.

[0047] As those skilled in the art will understand, typically when implementing the solution provided by the present invention using an active stereo device, two camera sensors will be used, while when implementing the solution provided by the present invention using a structured light device, one camera sensor will be used.

[0048] According to another embodiment, the at least one processor is configured to filter out projection patterns from processed data by applying an algorithm operable to detect clumps included in the at least one image. According to yet another embodiment, the at least one processor is configured to perform the following steps for each pixel included in a particular clump:

[0049] (a) Collect data relating to all pixels included within a square window surrounding that pixel;

[0050] (b) Remove all data from the collected information relating to pixels whose grayscale value is higher than the minimum value of all pixels included in the particular patch;

[0051] (c) Apply a bilateral filter to the collected data that was not removed in step (b) to fill the clumped traces with temporary values; and

[0052] (d) The processor is also configured to perform the following steps on the entire captured image:

[0053] (d1) Apply an operator to the obtained result, wherein the operator is configured to remove the projection pattern from the captured image while preserving the edge positions in the captured image; and

[0054] (d2) Apply edge detection operators to patternless data to detect various edges in the captured image.

[0055] According to another embodiment of this aspect of the present disclosure, the at least one processor is further configured to determine key points in an actively illuminated, patternless image. For example, key points are intersections of lines (e.g., straight lines) detected in the background of a projected pattern. Attached Figure Description

[0056] To gain a more complete understanding of the present invention, reference is now made to the following detailed description taken in conjunction with the accompanying drawings, wherein:

[0057] Figure 1 An example of a scene captured by a stereoscopic image capture device is shown;

[0058] Figure 2 Through the Figure 1 The image depicted in the image was obtained by applying a logarithmic transformation;

[0059] Figure 3 The example illustrates the result obtained by applying a bilateral filter to the collected data to obtain the fill value of the clumping, according to an embodiment explained in accordance with the present invention;

[0060] Figure 4 The final edge map is shown;

[0061] Figure 5 Examples of the Figure 1 A magnification of the region of interest within the illustrated image;

[0062] Figure 6 A second example of the input image is shown;

[0063] Figure 7 Examples Figure 6 The final edge map of the example depicted in the image;

[0064] Figure 8 It shows the Figure 6 Enlarging the area of ​​interest within; and

[0065] Figure 9 It shows the Figure 6 The area of ​​focus is magnified. Detailed Implementation

[0066] In this disclosure, the term "comprising" is intended to have an open-ended meaning, such that when a first element is stated to include a second element, the first element may also include one or more other elements that are not necessarily identified or described herein or recorded in the claims.

[0067] In the following description, numerous specific details are set forth for illustrative purposes in order to provide a better understanding of the invention by way of example. However, it should be understood that the invention can be practiced without these specific details.

[0068] Known patterns are typically projected onto the scene by structured light sensors or active stereo sensors. When appearing on a sensor camera, the deformation of these known patterns as they impact the surface allows the vision system to calculate depth and surface information of objects present in the scene, for example, as used in structured light 3D scanners.

[0069] A pattern projector may include a light source, one or more optical components, and a housing. The purpose of the device is to project light patterns. This is typically achieved by manipulating light emitted from a light source or array of light sources and shaping it into a desired pattern at a desired intensity. In many projector modules, lasers or laser arrays are used, and optionally, lenses and pattern-forming elements, microstructured optics (e.g., diffusers or DOEs) are employed.

[0070] The main problem addressed by this disclosure is the detection of edges in a captured image of a scene illuminated by an active stereo pattern projector or a structured light projector. The detection of these edges can help refine the edges in a depth map obtained using active stereo or structured light by incorporating them during stereo matching or as a post-processing cleaning stage.

[0071] Therefore, this invention proposes using the background intensity of the projector. Some projectors include this background intensity as a "built-in" function, while for other projectors that do not have this background intensity function, a floodlight projector can be used to add background intensity.

[0072] The main issues that must be addressed to solve this problem are as follows:

[0073] 1. Detection of both near and far edges.

[0074] 2. Ignore / eliminate the edges of spots, and

[0075] 3. If the projector pattern is "removed" when processing the captured image, it must be done in a way that does not introduce new edges into the image. Furthermore, any pattern removal process should preferably avoid moving the positions of background edges, or otherwise move them as little as possible.

[0076] As already explained, the basic idea of ​​the solution provided by this invention is to detect edges included in a stereoscopic image having a pattern generated by a pattern projector. Many methods can be used to implement this idea. For example, an intuitive approach could be to alternate between passive and active images, or between images including the projected pattern and images generated simultaneously by a floodlight projector, and then remove the edges from the images without any blob. However, an inherent drawback of this approach is that motion of objects may exist between consecutive frames, thus requiring the inclusion of memory for storing frames and the elimination of motion effects to implement the solution in this case. Clearly, implementing such a solution in a hardware system is quite complex.

[0077] Figure 1 An example of a scene captured by a stereoscopic image capture device is presented. This scene (i.e., the image) will be used in the following description for implementing the processing techniques explained according to embodiments of the invention.

[0078] Preferably, image processing for detecting edges and processing the image accordingly can be accomplished using an apparatus as claimed in this invention, comprising two image capture devices and a processor (e.g., on a chip). Alternatively, after an image is retrieved from the image capture devices, if the image capture devices include a processing unit, the processing can be performed within the image capture devices (camera devices) or via application software using a computer (e.g., a laptop computer, desktop computer, etc.). It should be understood that all these options are included within the scope of this invention.

[0079] Figure 2 Demonstrated by Figure 1The image depicted is obtained by applying a logarithmic transformation to detect the relative intensity of edges included in the image.

[0080] The solution provided by this invention is based on a general method of filtering out projection patterns. This is because most edges depicted in the image being processed belong to the projection pattern rather than the edges of the object (e.g., ...). Figure 1 (As shown), therefore, the filtering is performed. The filtering phase is performed by coordinating an appropriate algorithm with the projection pattern.

[0081] The following are examples of methods explained according to embodiments of this disclosure, in which a logarithmic transformation is first applied to an image to detect the intensity of edges with relative intensity, such as... Figure 2 As shown. Next, a segmentation algorithm is applied to the obtained results to detect blobular clumps in the pattern, such as the MSER (Maximally Stable Extremal Regions) algorithm. The selected algorithm should be one configured to perform clump detection in the image. The term "clump detection" as used throughout the specification and claims refers to a method designed to detect regions in a digital representation of an image that differ in properties (e.g., brightness or color) compared to their surroundings. Typically, a clump is a region in an image that has certain properties that are constant or approximately constant, and in a sense, all points within the clump can be considered similar to each other.

[0082] Next, fill in the pattern pixels in each pattern patch and find the minimum value of all pixels in the patch.

[0083] Now, for each clump pixel, perform the following steps:

[0084] (i) Collect information relating to all pixels included in a square window surrounding the pixel;

[0085] (ii) Remove all information from the collected data relating to pixels whose grayscale value is higher than the minimum of all pixels in the found clumps. This step aims to ensure that only non-patterned pixels are used;

[0086] (iii) Apply a bilateral filter to the collected information that was not removed in step (ii) to obtain the fill value. Figure 3 ).

[0087] Next, optionally, the provided method may also include the following steps performed on the entire captured image, as described below.

[0088] The obtained results are applied using an antiharmonic filter, for example, as disclosed in the following literature: Jesus Angulo, “Morphological bilateral filtering”, SIAM J. of Imaging Sciences, July 2013.

[0089] Alternatively, the grayscale operator is:

[0090]

[0091] If necessary when using the opening operator, the filter is preferably applied twice: once with -p (soft minimum, erosion) and once with +p (soft maximum, dilation) to achieve a "soft opening" operation on the image. This operation makes the blobs in the pattern disappear while preserving the edge positions.

[0092] Apply edge detection operators (e.g., a variant of the Canny edge detector) to patternless data to detect various edges in the captured image.

[0093] According to another embodiment of the invention, the provided method further includes the step of determining key points in the image being processed. Examples of such key points may be, for example, the intersection of straight lines in a patternless image.

[0094] In the specification and claims of this application, each of the verbs “comprising,” “including,” and “having,” and its conjugations, used to indicate the object (or multiple objects) of the verb, is not necessarily a complete list of members, components, elements, or parts of the subject (or multiple subjects) of the verb.

[0095] The invention has been described in detail using embodiments provided by way of example and not intended to limit the scope of the invention in any way. The described embodiments include various features, and not all features are necessary in all embodiments of the invention. Some embodiments of the invention utilize only a few features or possible combinations of features. Variations of the described embodiments of the invention, as well as embodiments of the invention including different combinations of features indicated in the described embodiments, will occur to those skilled in the art. The scope of the invention is limited only by the appended claims.

Claims

1. A method for generating a three-dimensional depth map, comprising the following steps: (i) Illuminate the target by a pattern projector with background intensity or by a combination of a floodlight projector and a pattern projector without background intensity operating together; (ii) Capturing at least one image, the at least one image comprising one or more objects present at the illuminated target, wherein the at least one image of at least one of the one or more objects comprises a blob; (iii) Convert the at least one captured image into data by applying a logarithmic transformation to the at least one captured image; (iv) The data received from the transformation of the at least one captured image is processed by applying a grayscale opening operator to the processed data using an antiharmonic filter, thereby eliminating blobes in the data associated with the projection pattern of the captured image while preserving the edge positions of the one or more objects included in the captured image. (v) Apply an edge detection operator to detect the edge of at least one of the one or more objects present at the illuminated target in patternless data associated with the captured image; as well as (vi) Generate a three-dimensional depth map, the three-dimensional depth map including at least one of the one or more objects whose edges have been determined.

2. The method according to claim 1, wherein, The step of detecting the edge of at least one of the one or more objects present at the illuminated target is performed by detecting non-patterned edges included in the at least one image.

3. The method according to claim 1, wherein, Step (iv) is performed by applying an algorithm configured to detect clumps included in the at least one image.

4. The method according to claim 3, wherein, Step (iv) also includes filling the pixels of the projected pattern belonging to each pattern clump with pixels, and determining the minimum value of all pixels included in the clump.

5. The method according to claim 4, further comprising: The following steps are for each pixel included in a specific clump: (a) Collect data relating to all pixels included within a square window surrounding the pixel; (b) Remove all data from the collected information relating to pixels whose grayscale value is higher than the minimum value of all pixels included in the particular patch; (c) Apply a bilateral filter to the collected data that was not removed in step (b).

6. The method according to claim 5, further comprising: The following steps are performed on the entire captured image: (d1) Apply an operator to the obtained result, wherein the operator is configured to remove the projection pattern from the captured image while preserving the edge positions in the captured image; as well as (d2) Apply edge detection operators to patternless data to detect various edges in the captured image.

7. The method according to claim 6, further comprising: Steps for determining key points in an actively illuminated, patternless image.

8. The method according to claim 7, wherein, The key point is the intersection of lines detected in the background of the projected pattern.

9. An apparatus for generating a three-dimensional depth map, wherein, The device includes: At least one projector, which is a pattern projector with background intensity or a combination of a floodlight projector and a pattern projector without background intensity configured to operate together, is suitable for illuminating a target while projecting a pattern at the target, where one or more objects are present. At least one image capture sensor is configured to capture an image comprising one or more objects present at an illuminated target, wherein the at least one image of at least one of the one or more objects comprises a blob; and At least one processor is configured to operate on images captured by the at least one image capture sensor, and is configured to: (a) Processing the at least one captured image by applying a logarithmic transformation to the at least one captured image and converting the at least one captured image into processed data; (b) Applying a grayscale opening operator to the processed data using an antiharmonic filter, thereby eliminating blobes in the data associated with the projection pattern of the captured image while preserving the edge positions of the one or more objects included in the captured image; (c) Determine the edge of at least one of the one or more objects present at the illuminated target in patternless data associated with the captured image; and (d) Generate a three-dimensional depth map, the three-dimensional depth map including at least one object whose edges have been determined among the one or more objects.

10. The apparatus according to claim 9, wherein, The at least one processor is configured to filter out the projection pattern from the processed data.

11. The apparatus according to claim 10, wherein, The at least one processor is configured to detect the edge of at least one of the one or more objects present at the illuminated target by detecting non-patterned edges included in the at least one image.

12. The apparatus according to claim 9, wherein, The at least one processor is configured to filter out projection patterns from the processed data by applying an algorithm that operates to detect clumps included in the at least one image.

13. The apparatus according to claim 12, wherein, The at least one processor is configured to fill the pixels of the projected pattern belonging to each pattern clump, and to determine the minimum value of all pixels included in the clump.

14. The apparatus according to claim 9, wherein, The at least one processor is also configured to determine key points in an actively illuminated, patternless image.

15. The apparatus according to claim 14, wherein, The key point is the intersection of lines detected in the background of the projected pattern.

Images