Imaging device with three-dimensional information measurement function, three-dimensional information measurement method, and image acquisition method with three-dimensional information

Abstract

Estimating depth regardless of the imaged object or imaging conditions. [Solution] The imaging device 100 with a three-dimensional information measurement function comprises a pattern projection unit 2 having a light source 20 and a pattern forming unit 21 for forming a projection pattern of the light rays; an imaging unit 1 having a lens group 10 including at least one lens, an encoding aperture 11 for narrowing the ambient light passing through the lens group 10 with a predetermined encoding pattern, a shutter 12, and an image sensor 13; and a depth calculation unit 3 that calculates the depth distribution of the image captured by the imaging unit 1 from a pattern image captured by the imaging unit 1 and a known blur function related to the encoding pattern.

Description

Technical Field

[0006] , ,

[0001] The present invention relates to an imaging device with a three-dimensional information measurement function, a three-dimensional information measurement method, and an image imaging method with three-dimensional information.

Background Art

[0002] Patent Document 1 describes an explanation of an encoded imaging method. The encoded imaging method uses a mask of a complex pattern as an aperture (encoded aperture), and by controlling the shape and frequency characteristics of a general blur function (PSF), it is described that a method called Depth from defocus (DFD) that estimates the depth of a scene from the blur of an image can be used.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] DFD estimates the depth of the imaged scene from the blur of the captured image. However, if the captured image cannot be restored from the blur in the first place, such an estimate is impossible. Therefore, when there is no change over the entire image or a considerable range, for example, when a flat wall surface is the imaging object, or when the image is dark and the subject hardly appears in the image, it is impossible to estimate the depth of the scene.

[0005] The present invention has been made in view of such circumstances, and its object is to estimate the depth regardless of the imaging object or imaging conditions.

Means for Solving the Problems

[0006] To solve the above problems, the imaging device with three-dimensional information measurement function according to this application comprises: a pattern projection unit having a light source and a pattern forming unit for forming a projection pattern of the light rays; an imaging unit having a lens group including at least one lens, an encoding aperture for narrowing the ambient light passing through the lens group with a predetermined encoding pattern, a shutter, and an image sensor; a depth calculation unit that determines the depth distribution of the image captured by the imaging unit from a pattern image captured by the imaging unit and a known blur function related to the encoding pattern.

[0007] Furthermore, in order to solve the above problems, the three-dimensional information measurement method according to this application projects a predetermined pattern onto an object using light rays, images the object onto which the predetermined pattern is projected through an encoding aperture having a predetermined encoding pattern, and determines the depth distribution of the pattern image from the captured pattern image and a known blur function related to the encoding pattern.

[0008] Furthermore, in order to solve the above problems, the three-dimensional information image acquisition method according to this application projects a predetermined pattern onto an object using light rays, images the object onto which the predetermined pattern is projected by an imaging unit through an encoding aperture having a predetermined encoding pattern, determines the depth distribution of the pattern image from the captured pattern image and a known blur function related to the encoding pattern, captures a visible light image of the object using visible light rays with the imaging unit, and adds the depth distribution to the visible light image. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of an imaging device with a three-dimensional information measurement function according to a preferred embodiment of the present invention. [Figure 2] This is a schematic diagram showing how an imaging device with three-dimensional information measurement capabilities is capturing images of an object being imaged. [Figure 3] This figure shows an invisible light image captured by the imaging unit in the example shown in Figure 2. [Figure 4]This diagram shows the principle of depth estimation using an encoded aperture. [Figure 5] This figure shows examples of various projection patterns. [Figure 6] This flowchart illustrates the procedure for a three-dimensional information measurement method and a three-dimensional information image acquisition method using an imaging device with a three-dimensional information measurement function according to this embodiment. [Modes for carrying out the invention]

[0010] In this application, the drawings may schematically represent the width, thickness, shape, etc., of each part in order to clarify the explanation, but these are merely examples and do not limit the interpretation of the present invention. In this specification and in each drawing, elements having the same function as those described with respect to previously shown drawings are denoted by the same reference numerals, and redundant explanations may be omitted.

[0011] Furthermore, in the detailed description of the present invention, when defining the positional relationship between one component and another component, "above" and "below" include not only cases where the component is located directly above or directly below another component, but also cases where other components are interposed between them, unless otherwise specified.

[0012] Figure 1 is a schematic diagram of an imaging device 100 with a three-dimensional information measurement function according to a preferred embodiment of the present invention. The imaging device 100 with a three-dimensional information measurement function comprises an imaging unit 1, a pattern projection unit 2, and a depth calculation unit 3.

[0013] In this embodiment, the imaging unit 1 has the configuration of a so-called digital camera, and is configured to house a lens group 10 including at least one lens, an encoded aperture 11 that narrows the ambient light passing through the lens group according to a predetermined encoded pattern, a shutter 12, and an image sensor 13 within a housing 14.

[0014] The pattern projection unit 2 has a configuration in which a light source 20 that emits invisible light rays, a pattern forming unit 21 that forms a projection pattern of invisible light rays, and a projection lens 22 are also housed in a housing 23.

[0015] The depth calculation unit 3 is an information processing device that processes image data obtained from the image sensor 13 of the imaging unit 1 to determine the depth distribution of the captured image.

[0016] The depth calculation unit 3 may be configured separately from the imaging unit 1 and be able to communicate with each other via wired or wireless means, or it may be integrated with the imaging unit 1. Furthermore, the imaging unit 1 and the pattern projection unit 2 do not necessarily have to be in separate housings 14 and 23 as shown in Figure 1, but may be integrated. Note that detailed configurations such as the power supply circuit that supplies power to the imaging unit 1, the pattern projection unit 2, and the depth calculation unit 3, the controller that controls them, the user interface including buttons for operating the imaging device 100 with three-dimensional information measurement function, the I / O for inputting and outputting information with external devices, the electronic circuits such as memory and processors, and other details are not necessarily required for describing the imaging device 100 with three-dimensional information measurement function, and therefore their illustration and detailed explanation are omitted.

[0017] The lens group 10 may be a set of imaging lenses commonly used in cameras, and may be capable of adjusting the focal length, depth of field, and zoom magnification as appropriate. The material, coating, number of groups, and number of lenses constituting the lens group 10 are not particularly limited, and the lens group 10 may be a fixed-focus single lens. The adjustment of the lens group 10 may be performed automatically or manually.

[0018] The encoded aperture 11 partially shields the external light passing through the lens group 10 with a specific mask pattern. Specific examples of the encoded aperture 11 can be a black plate with an aperture of a specific pattern shape, or a transparent plate such as glass with a specific black pattern printed on its surface. Furthermore, the encoded aperture 11 can be a liquid crystal shutter, and the aperture pattern can be changed. For example, the presence or absence of the encoded aperture can be switched by switching the display and non-display of a specific mask pattern. Alternatively, the type of the encoded aperture can be changed by switching multiple types of mask patterns. Furthermore, as the liquid crystal shutter used for the encoded aperture 11, a dot matrix type liquid crystal display can be used to display or not display an arbitrary mask pattern, and also display a normal aperture pattern of a circular aperture.

[0019] The shutter 12 is a component that functions as a shutter of a normal camera and adjusts the exposure amount of external light to the image sensor 13. The shutter 12 may use a general mechanical shutter, but in this embodiment, a liquid crystal shutter is used. The shutter 12 includes an infrared shutter 12a that can switch between transmission and non-transmission of infrared rays and a visible light shutter 12b that can switch between transmission and non-transmission of visible light. Since the infrared rays used in this embodiment are infrared rays, the infrared shutter 12a is an infrared shutter, but when ultraviolet rays are used as the infrared rays, the infrared shutter 12a may be an ultraviolet shutter.

[0020] Then, when both the infrared light shutter 12a and the visible light shutter 12b that make up the shutter 12 are closed, the external light indicated by the dashed-dotted line A in FIG. 1 is blocked and does not reach the image sensor 13. On the other hand, when the infrared light shutter 12a is open, infrared light within the external light A reaches the image sensor 13. Conversely, when the visible light shutter 12b is open, visible light within the external light A reaches the image sensor 13. Therefore, the shutter 12 is a member that selectively allows infrared light (infrared light in this embodiment) and visible light to pass through.

[0021] The image sensor 13 is a two-dimensional optical sensor capable of detecting infrared light and visible light. There is no limitation on the type of the image sensor 13, and it may be a general CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device). Also, the detection elements arranged on the image sensor 13 may be arranged with both those suitable for detecting infrared light and those suitable for detecting visible light, or may be those that simultaneously detect without particularly distinguishing the user. The image detected by the image sensor 13 may be a color image or a monochrome image.

[0022] The light source 20 is a light source that emits infrared light, that is, light outside the visible region, and is an infrared light source in this embodiment. And since the light source 20 is also a laser light source in this embodiment, it is an infrared laser oscillator. The infrared light emitted from the light source 20 is shaped using an appropriate optical system as needed and is incident on the pattern forming unit 21 as shown by the dashed-dotted line B. Note that the light source 20 may be an ultraviolet light source.

[0023] The pattern forming unit 21 is a component that forms a projection pattern, which is a predetermined pattern when projecting invisible light emitted from the light source 20 to the outside. In this embodiment, it is a pattern mirror. That is, a specific pattern is engraved on the surface of the mirror, and the reflected light forms the projection pattern. In addition, the pattern forming unit 21 may be, for example, a DMD (Digital Micromirror Device), and any projection pattern may be obtained by controlling the DMD. Furthermore, the pattern forming unit 21 may be a scanning optical system using a polygon mirror, and any projection pattern may be projected by scanning by controlling the oscillation pattern of invisible light from the light source 20 or the switching timing of shielding.

[0024] The projection lens 22 is an optical system that projects patterned light, indicated by the dashed-dotted line C, which is reflected by the pattern forming section 21, outwards. Figure 1 shows a configuration with a single lens, but it may also consist of multiple lenses, or be capable of adjusting the focal length, projection magnification, etc. The projected light rays are magnified as appropriate, as shown by the dashed-dotted line D.

[0025] Figure 2 is a schematic diagram showing how the imaging device 100 with three-dimensional information measurement function is imaging an object 4 that is the target of imaging. The nature of the object 4 varies, but as shown in Figure 2, if the surface is flat and there is little texture such as color variation, the image captured by the imaging unit 1 will have almost no change between the pixels that make up the image, and blur cannot be detected.

[0026] As shown in Figure 2, the pattern projection unit 2 projects a projection pattern 40 using invisible light (infrared light in this embodiment) onto the surface of the object 4. In the illustrated example, the projection pattern 40 is a grid pattern. When the imaging unit 1 captures an invisible light image with the projection pattern 40 using invisible light projected onto the surface of the object 4, the blur on the surface of the object 4 can be observed using the projection pattern 40, and therefore, depth estimation using an encoded aperture becomes possible.

[0027] Figure 3 shows an invisible light image captured by the imaging unit 1 in the example shown in Figure 2. In the vicinity of the focal length determined by the lens group 10 of the imaging unit 1, the image is in focus, and the projection pattern 40 is clearly visible near the center of the invisible light image. However, as the surface of the object 4 moves away from the focal length, the projection pattern 40 becomes blurred as it approaches the left and right edges of the invisible light image.

[0028] Since the depth calculation unit 3 estimates the depth from the blur of this invisible light image, it is desirable that the projection pattern 40 is projected clearly on the surface of the object 4, regardless of the distance between the object 4 and the imaging device 100 with three-dimensional information measurement function. For this reason, it is desirable that the projection pattern 40 projected from the pattern projection unit 2 is not an image formed on the surface of the object 4, but simply an amplified projection of collimated light. That is, the patterned light shown by the dashed line C in the pattern projection unit 2 of Figure 1 is collimated light. For this reason, it is desirable that the light source 20 be an appropriate collimated light source, and in this embodiment, it is a laser light source. Of course, if the light source 20 is not a laser light source, it may be any light source and an appropriate collimator.

[0029] Furthermore, as is clear from Figure 3, since the depth calculation unit 3 estimates depth from the blur of the invisible light image, the depth estimation is performed based on the focal length of the optical system of the imaging unit 1. Therefore, the focal length of the lens group 10 when the imaging unit 1 captures the invisible light image must be known. If the lens group 10 is a fixed-focus lens, the focal length of that lens is used. If the lens group 10 is a variable-focus lens, the focal length at the time of imaging is used, or the arrangement of lenses included in the lens group 10 whose focal length is known in advance is used at the time of imaging.

[0030] Figure 4 shows the principle of depth estimation using the coded aperture 11. The figure schematically shows how ambient light passing through the coded aperture 11 is refracted by the lens group 10 (shown here as a single lens for simplicity) and strikes the image sensor 13. In the figure, (a) shows the optical path of a light ray from the surface of the object 4 that is closer than the focal length of the lens group 10, and (b) shows the optical path of a light ray from the surface of the object 4 that is further than the focal length of the lens group 10, both shown by dashed lines.

[0031] In the examples of Figure 4(a) and (b), the light rays from the surface of the object 4 do not form an image on the image sensor 13, but are captured as blurred. In this case, as shown in (a), the light rays encoded by passing through the encoding aperture 11 enter the image sensor 13 in the orientation of encoding shown in 5a, without changing the geometric positional relationship of the encoding.

[0032] For convenience, Figure 4 shows the encoding pattern of the encoding aperture 11 projected onto the surface of the image sensor 13 in encoding direction 5a. However, this is not actually the case. The figure shows that the spatial frequency characteristics of the blur of light rays entering the image sensor 13 follow the PSF corresponding to the encoding pattern shown in encoding direction 5a. Hereafter, this geometric positional relationship will be referred to as the forward direction, and the PSF in this case will be referred to as the forward PSF.

[0033] In contrast, in the case shown in (b), the light rays encoded by passing through the encoding aperture 11 enter the image sensor 13 with the encoding orientation shown in 5b, with the geometric positional relationship of the encoding inverted vertically and horizontally. The encoding orientation 5b in Figure 4, like encoding orientation 5a, indicates that the spatial frequency characteristics of the blur of the light rays entering the image sensor 13 follow the PSF corresponding to the encoding pattern shown in encoding orientation 5b. Hereafter, this geometric positional relationship will be referred to as the reverse direction, and the PSF in this case will be referred to as the reverse direction PSF.

[0034] In this case, if the forward PSF and the reverse PSF are the same, it is not possible to determine from the blur of the image captured by the image sensor 13 whether the surface of the object 4 is far or near relative to the focal length of the lens group 10. On the other hand, since the coding direction 5a and the coding direction 5b are in a positional relationship rotated 180 degrees with respect to the optical axis, if the coding pattern in the coding aperture 11 is a figure that can distinguish between these two, that is, a double-asymmetric figure as exemplified in Figure 4, then the forward PSF and the reverse PSF will be different, and the distance of the surface of the object 4 relative to the focal length of the lens group 10 can also be determined using each PSF.

[0035] In the example described above, the projection pattern 40 projected from the pattern projection unit 2 was a grid pattern, but the specific shape of the projection pattern 40 can be arbitrary. Figure 5 shows examples of various projection patterns 40, where (a) is the grid pattern already described, (b) is a dot matrix pattern, and (c) is a houndstooth pattern. The projection pattern 40 can be selected according to the surface material of the object 4, the expected distance from the object 4, etc., or the pattern projection unit 2 can be configured to switch between projection patterns.

[0036] Figure 6 is a flowchart illustrating the procedure for a three-dimensional information measurement method and a three-dimensional information image acquisition method using the imaging device 100 with three-dimensional information measurement function according to this embodiment.

[0037] First, in step ST1, a predetermined pattern is projected onto the object 4 using invisible light from the pattern projection unit 2. As a result, as illustrated in Figure 2, the surface of the object 4 is covered with a projection pattern 40 made of invisible light.

[0038] In the subsequent step ST2, the imaging unit 1 images the object 4 onto which the projection pattern 40 is projected, through the encoding aperture 11. At this time, the imaging unit 1 shown in Figure 1 is controlled so that a predetermined specific mask pattern is displayed if the encoding aperture 11 is capable of changing aperture patterns. The lens group 10 is maintained in a setting suitable for photographing the object 4, and its focal length is known. The shutter 12 operates so that the invisible light shutter 12a is opened for a predetermined time to expose the image sensor 13 to ambient light, while the visible light shutter 12b is closed, allowing the image sensor 13 to capture only invisible light. As a result, a pattern image is obtained by the image sensor 13.

[0039] Furthermore, in step ST3, the depth calculation unit 3 performs deconvolution on the pattern image obtained by the image sensor 13 using a known PSF (blur function) to estimate the depth of each part of the pattern image. The depth of each part of the pattern image thus obtained is an image that shows the distribution of depth within the pattern image plane, and this can be called the depth distribution.

[0040] Steps ST1 to ST3 described above measure the depth distribution of object 4, that is, the three-dimensional information of object 4. This information can be used, for example, for various measurements or modeling of object 4. Therefore, steps ST1 to ST3 constitute a three-dimensional information measurement method.

[0041] Furthermore, in step ST4, the imaging unit 1 captures a visible light image of the object 4 using visible light. At this time, if the encoding aperture 11 shown in Figure 1 can change the aperture pattern, the imaging unit 1 may hide a specific mask pattern for encoding or display a normal circular aperture pattern. If the encoding aperture 11 displays a fixed pattern, imaging of the object 4 using visible light is possible even if the specific mask pattern is displayed. The lens group 10 is also maintained in a setting suitable for photographing the object 4. The shutter 12 operates so that the invisible light shutter 12a is closed to prevent the projection pattern 40 from being captured by the image sensor 13, while the visible light shutter 12b is opened for a predetermined time to expose the image sensor 13 to a visible light image of the object 4. As a result, the image sensor 13 captures a natural visible light image of the object 4.

[0042] Step ST4 does not necessarily have to be performed after steps ST2 and ST3, and may be performed before or after step ST2. If the imaging unit 1 is not fixed, it is desirable that steps ST2 and ST4 be performed consecutively, that is, virtually simultaneously, so that the pattern image and the visible light image are captured almost simultaneously. This is because the pattern image and the visible light image capture the object 4 from the same angle, the positional relationship of the object 3 in both images is the same, and they can be superimposed on each other.

[0043] Finally, in step ST5, the depth distribution obtained in step ST3 is added to the visible light image obtained in step ST4. Specifically, this can be achieved by adding Z information, which indicates depth, to each pixel in a visible light image that has RGB information (or information according to an appropriate color system such as CMY) for each pixel, or by adding an image showing the depth distribution to a visible light image that consists of a set of color images such as RGB.

[0044] As a result, steps ST1 to ST5 capture an image to which the depth distribution of object 4 is added, i.e., an image to which three-dimensional information of object 4 is added. Since such an image is originally a two-dimensional image with additional depth information added, it can be used for various purposes such as measurement, modeling, and object recognition, and can be applied to various applications. Therefore, steps ST1 to ST5 constitute a method for capturing images with three-dimensional information.

[0045] In the embodiment described above, the light source 20 emits invisible light, and by making the light emitted from the pattern projection unit 2 invisible light, the pattern projected onto the object 4 when measuring the three-dimensional information of the object 4 can be made invisible. However, the invention is not limited to this, and if it is acceptable to project a pattern using visible light onto the object 4, the light source 20 may be made to emit visible light, and the pattern projection unit 2 may project a pattern using visible light. In this case, since visible light is used for the projection pattern, the invisible light shutter 12a becomes unnecessary, and the imaging unit 1 can be made lighter. [Explanation of symbols]

[0046] 1 Imaging unit, 2 Pattern projection unit, 3 Depth calculation unit, 4 Object, 5a, 5b Encoding orientation, 10 Lens group, 11 Encoding aperture, 12 Shutter, 12a Invisible light shutter, 12b Visible light shutter, 13 Image sensor, 14 Housing, 20 Light source, 21 Pattern molding unit, 22 Projection lens, 23 Housing, 40 Projection pattern, 100 Imaging device with three-dimensional information measurement function.

Claims

1. A pattern projection unit having a light source that emits light rays and a pattern forming unit that forms a projection pattern of the light rays, An imaging unit having a lens group including at least one lens, an encoded aperture that restricts the ambient light passing through the lens group according to a predetermined encoded pattern, a shutter, and an image sensor, A depth calculation unit that calculates the depth distribution of the image captured by the imaging unit from a pattern image captured by the imaging unit and a known blur function related to the encoded pattern, An imaging device with a three-dimensional information measurement function.

2. The imaging device with three-dimensional information measurement function according to claim 1, wherein the light source is a laser light source.

3. The imaging device with three-dimensional information measurement function according to claim 1, wherein the aperture pattern of the encoded aperture is a double-asymmetric figure.

4. The imaging device with three-dimensional information measurement function according to claim 1, wherein the encoding aperture is a liquid crystal shutter and the aperture pattern can be changed.

5. The light emitted by the aforementioned light source is invisible light. The image sensor is capable of detecting invisible and visible light. The imaging device with three-dimensional information measurement function as described in claim 1.

6. The imaging device with three-dimensional information measurement function according to claim 5, wherein the shutter includes an invisible light shutter capable of switching between transmitting and blocking invisible light, and a visible light shutter capable of switching between transmitting and blocking visible light.

7. The imaging device with three-dimensional information measurement function according to claim 6, wherein the invisible light is infrared light.

8. A predetermined pattern is projected onto the object using light rays. The object onto which the predetermined pattern is projected is imaged through an encoding aperture having a predetermined encoding pattern. The depth distribution of the pattern image is determined from the captured pattern image and a known blur function related to the encoded pattern. Three-dimensional information measurement method.

9. The three-dimensional information measurement method according to claim 8, wherein the light ray is an invisible light ray.

10. A predetermined pattern is projected onto the object using light rays. The object onto which the predetermined pattern is projected is imaged by the imaging unit through an encoding aperture having a predetermined encoding pattern. The depth distribution of the pattern image is determined from the captured pattern image and the known blur function related to the encoded pattern. The imaging unit captures a visible light image of the object using visible light, The depth distribution is added to the visible light image. A method for acquiring images with three-dimensional information.

11. The method for capturing a three-dimensional image with information according to claim 10, wherein the light ray is invisible light.

Images