Imaging device, subject depth estimation method, and program

Abstract

The present invention improves practicality in subject depth estimation through encoded image-capturing.　Provided is a subject depth estimation method which involves: acquiring captured unmasked images by capturing a subject in a state where a mask is not present; determining a representative edge direction on the basis of an edge image included in the captured unmasked image; selecting, from among a plurality of masks prepared in advance, a combination of masks of which depth estimation accuracies for an object, which corresponds to an image that represents an edge in the same direction as the representative edge, are relatively the highest; using each mask included in the selected combination of masks to capture the subject and acquire a plurality of captured masked images; and calculating depth estimation values at a plurality of positions in the subject by performing, on the plurality of captured masked images, decoding processes based on point spread functions that are unique to the selected masks, respectively.

Description

Technical Field

[0001] The present invention relates to an imaging device, a subject depth estimation method, and a program.

Background Art

[0002] In the field of coded imaging, a technique called DFD (Depth From Defocus) is known. The DFD technique is a technique for estimating the distance from the optical system of an imaging device to a subject, that is, the depth or depth of field of the subject, based on the degree of blurring of the edges captured in the image obtained by imaging.

[0003] The DFD technique is described, for example, in Non-Patent Document 1. In the DFD technique described in Non-Patent Document 1, two masks with different positions of the apertures through which light passes are prepared. Next, for each of the two masks, coded imaging is performed in which the mask is placed in the light incident region of the optical system to image the same subject. Then, decoding processing based on the point spread function unique to each mask is performed on the two captured images obtained by the coded imaging, and the depth of the subject is estimated. The point spread function is generally called a PSF (Point Spread Function), and is also referred to as a blurring function, a blurring spread function, a point image distribution function, and the like.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The DFD technology is still in the development stage, and there is much room for improvement in terms of practicality. Due to the above circumstances, a more practical DFD technology is desired.

Means for Solving the Problem

[0006] Among the inventions disclosed in the present application, an outline of a representative one will be described as follows.

[0007] A representative embodiment of the invention of the present application includes an optical system into which light from a subject is incident, an image sensor that receives the light passing through the optical system, a mask installation unit that creates a state in which any one of a plurality of pre-prepared masks is installed and a state in which none of the masks is installed with respect to an incident region of the light incident from the subject to the optical system, and an arithmetic control unit that outputs a signal for controlling the mask installation unit and the image sensor so that the subject is imaged, and acquires an image of the subject without a mask and an image of the subject with a mask. The arithmetic control unit performs an imaging process without a mask to obtain an image of the subject without a mask by imaging the subject without the mask, a determination process for determining a representative edge direction based on an edge image included in the image of the subject without a mask, a selection process for selecting a combination of the masks that has the relatively highest depth estimation accuracy for an object corresponding to an image representing an edge in the same direction as the representative edge direction from among the plurality of pre-prepared masks, an imaging process with a mask to obtain a plurality of images of the subject with a mask by imaging the subject using the individual masks included in the selected combination of masks, and a decoding process for obtaining information representing the depth at a plurality of positions of the subject by performing decoding on the plurality of images of the subject with a mask based on a point spread function unique to the selected mask. It is an imaging device that performs the above processes.

[0008] In a typical embodiment of the present invention, a maskless imaging image is obtained by imaging a subject without a mask, a representative edge direction is determined based on an edge image included in the maskless imaging image, and from among a plurality of masks prepared in advance, the combination of masks having the relatively highest accuracy in estimating the depth of an object corresponding to an image representing an edge in the same direction as the representative edge direction is selected. A plurality of imaging images with masks are obtained by imaging the subject using the individual masks included in the selected combination of masks, and information representing the depth at a plurality of positions of the subject is obtained by performing decoding based on a point spread function specific to the selected mask on the plurality of imaging images with masks. This is a method for estimating the depth of a subject.

[0009] In a typical embodiment of the present invention, a computer is caused to perform a maskless imaging process for obtaining a maskless imaging image by imaging a subject without a mask, a determination process for determining a representative edge direction based on an edge image included in the maskless imaging image, a selection process for selecting, from among a plurality of the masks prepared in advance, the combination of masks having the relatively highest accuracy in estimating the depth of an object corresponding to an image representing an edge in the same direction as the representative edge direction, a mask imaging process for obtaining a plurality of imaging images with masks by imaging the subject using the individual masks included in the selected combination of masks, and a decoding process for obtaining information representing the depth at a plurality of positions of the subject by performing decoding based on a point spread function specific to the selected mask on the plurality of imaging images with masks. This is a program for causing the computer to perform these processes.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Embodiments for Carrying Out the Invention

[0011] Before explaining each embodiment of the present invention, the basic content of the DFD technology and the problems discovered by the present inventors will be explained.

[0012] The state of blurring of a captured image (hereinafter referred to as "blur") generally depends on the point spread function determined by the optical system of the imaging device, the shape of the light incident region of the optical system, etc. When a mask that partially shields light is installed in the light incident region of the optical system, the point spread function is determined for each mask. Imaging a subject with an imaging device equipped with a mask is called coded imaging. When a subject is coded-imaged, a blurred image is obtained based on the point spread function unique to the mask used.

[0013] When a decoding process of performing inverse convolution based on the point spread function unique to the mask used is performed on the blurred image, a decoded image with improved blur and depth information of the object corresponding to each position of the decoded image can be obtained.

[0014] FIG. 17 is a diagram showing an example of a plurality of masks used in the DFD technique. Non-Patent Document 1 describes a DFD technique by coded imaging using the two masks Z1 and Z2 shown in FIG. 17. In the two masks Z1 and Z2 shown in FIG. 17, the black regions are regions that shield the light incident on the optical system, and the white regions are regions of openings that allow the light incident on the optical system to pass through. Thus, the two masks Z1 and Z2 shown in FIG. 17 have different geometric patterns of the openings through which light can pass.

[0015] On the other hand, as a result of studying the DFD technique by coded imaging using a plurality of masks with different geometric patterns of the openings, the present inventors found that there is an image in which the depth estimation accuracy is low among the images of the objects captured in the captured image. Specifically, an image with low depth estimation accuracy is an edge image representing an edge along a direction close to the displacement direction of the opening of the mask used.

[0016] An edge image representing an edge along the displacement direction of the openings of two masks used in DFD technology hardly changes in the degree of blurring between the captured image by the first mask and the captured image by the second mask. In the case of an image where the degree of blurring does not change between the two captured images, it is considered that the depth information cannot be accurately extracted even if the decoding process by inverse convolution based on the point spread function unique to each mask is performed.

[0017] Therefore, when the edge direction of a representative edge of interest among the edges in the captured image is the same as or close to the edge direction where there is almost no difference in the degree of blurring, the depth of the object corresponding to the edge image to be emphasized cannot be estimated with high accuracy.

[0018] Due to the above circumstances, in the DFD method of imaging the same subject using a plurality of masks and estimating the depth of the subject, a technique is desired that can estimate the depth of the corresponding object with more stable and high accuracy for the edge image in the representative edge direction in the captured image.

[0019] In view of the above circumstances, the present inventors have devised the present invention as a result of intensive studies. Hereinafter, each embodiment of the present invention will be described. Note that each embodiment described below is an example for carrying out the present invention and does not limit the technical scope of the present invention. Also, in each of the following embodiments, components having the same function are denoted by the same reference numerals, and repeated descriptions thereof are omitted unless particularly necessary.

[0020] (Embodiment 1) A subject depth estimation method according to Embodiment 1 of the present application will be described. The subject depth estimation method according to Embodiment 1 of the present application obtains an image without a mask by imaging a subject without a mask, determines a representative edge direction based on an edge image included in the image without a mask, and selects, from a plurality of masks prepared in advance, a combination of masks in which the depth estimation accuracy of an object corresponding to an image representing an edge in the same direction as the representative edge direction is relatively the highest. A plurality of images with masks are obtained by imaging the subject using the individual masks included in the selected mask combination, and information representing the depth at a plurality of positions of the subject is obtained by performing decoding based on a point spread function unique to the selected mask on the plurality of images with masks. This is the subject depth estimation method. The details of this subject depth estimation method are as follows.

[0021] FIG. 1 is a diagram showing an example of an imaging system and an arithmetic control device used in the subject depth estimation method according to Embodiment 1.

[0022] As shown in FIG. 1, the imaging system 80 has an optical system 81, an imaging device 82, and a mask M. The optical system 81 is mainly composed of a lens or the like, condenses the light L coming from the subject 90, and forms an image on the light receiving surface of the imaging device 82. The imaging device 82 is an electronic component that performs photoelectric conversion, photoelectrically converts the brightness of the image formed on the light receiving surface into the amount of charge by light, instantaneously captures the photoelectrically converted electrical signal, and obtains an imaging image.

[0023] The mask M is a so-called optical filter that allows a part of the light L incident on the optical system 81 to pass through and shields the other part of the light L. The mask is also called an encoding aperture, an encoding diaphragm, an aperture, or the like.

[0024] The arithmetic control device 91 is connected to the imaging device 82. The arithmetic control device 91 performs various processes on the imaging image obtained based on the electrical signal from the imaging device 82 and obtains various information. The arithmetic control device 91 is, for example, a computer.

[0025] FIG. 2 is a flowchart showing an example of the process flow of the subject depth estimation method according to Embodiment 1.

[0026] As shown in FIG. 2, in step H1, a plurality of masks are prepared. That is, for each of a plurality of specific edge directions (hereinafter also referred to as specific edge directions), a combination of masks M with high object depth estimation accuracy corresponding to an edge image representing the edge in that specific edge direction is possible, and a plurality of masks M are prepared.

[0027] Here, the mask M has a main aperture as a region through which the light L can pass with respect to the incident region of the light L incident on the optical system 81 used for imaging the subject 90 from the subject 90. Further, the plurality of masks M are two or more masks with different positions of the above-mentioned main apertures.

[0028] Also, here, as the specific edge directions, four directions of the vertical direction, the horizontal direction, the 45-degree right diagonal direction, and the 45-degree left diagonal direction are defined. The plurality of masks M to be prepared are, for example, the four masks M1 to M4 shown in FIG. 1. The masks M1 to M4 have a form in which circular light shielding regions for shielding light are formed in the upper right, upper left, lower left, and lower right, respectively, with respect to a shielding plate that shields the light incident region of the optical system 81. Details of an example of the plurality of masks to be prepared will be described later.

[0029] In step H2, imaging of the subject without a mask is performed. That is, the subject 90 is imaged using the optical system 81 and the image sensor 82 in a state where no mask is installed, and an image P0 without a mask is obtained. Note that the control of the optical system 81 or the image sensor 82 for imaging without a mask is performed by, for example, the arithmetic control device 91.

[0030] In step H3, a representative edge direction in the maskless imaging image is determined. That is, based on the edge image included in the maskless imaging image P0, a representative edge direction (hereinafter also referred to as the representative edge direction) that is considered important in the maskless imaging image P0 is determined. The representative edge direction is determined, for example, by selecting from among a plurality of predetermined specific edge directions. Details of an example of a method for determining the representative edge direction will be described later. The determination of the representative edge direction is performed, for example, by the arithmetic control device 91.

[0031] In step H4, a selection of a combination of masks to be used for masked imaging is made. That is, from among a plurality of masks M prepared in advance, a combination of masks M that has the relatively highest depth estimation accuracy for an object corresponding to an edge image representing an edge in the same direction as the determined representative edge direction is selected. In other words, among the combinations of masks M, a combination of masks M in which a specific edge direction with high depth estimation accuracy coincides with or approximates the determined representative edge direction is selected as the combination of masks M to be used for encoded imaging.

[0032] Note that the combination of masks M that has the relatively highest depth estimation accuracy for an object corresponding to an image representing an edge in the same direction as the representative edge direction is two or more masks in which the displacement direction of each main aperture or light-shielding region is orthogonal to the representative edge direction.

[0033] In step H5, masked imaging of the subject using the first mask is performed. That is, with the first mask included in the selected combination of masks installed, masked imaging of the same subject 90 using the optical system 81 and the imaging device 82, so-called encoded imaging, is performed. By performing this masked imaging, a first masked imaging image P1 is obtained.

[0034] In step H6, masked imaging of the subject is performed using the second mask. That is, masked imaging of the same subject 90 using the optical system 81 and the imaging device 82, so-called encoded imaging, is performed with the second mask included in the selected mask combination installed. By performing this masked imaging, a second masked imaging image P2 is obtained. Note that the control of the optical system 81 or the imaging device 82 for masked imaging is performed by, for example, the arithmetic control device 91.

[0035] In step H7, decoding processing of the masked imaging image is performed. That is, the first masked imaging image P1 and the second masked imaging image P2 are decoded by inverse convolution based on the point spread function unique to each of the two masks used. By performing this decoding processing, a decoded image with improved blurring of the subject is obtained, and information capable of estimating the depth of the object corresponding to each position of the decoded image is obtained. Note that the decoding processing is performed by, for example, the arithmetic control device 91.

[0036] Note that the point spread function unique to the mask used is determined by the geometric pattern of the aperture or light-shielding region of the mask, the configuration of the optical system, the configuration of the imaging device, the positional relationship between the mask, the optical system, and the imaging device, and the like. Further, the decoding processing in step H7 may be, for example, the decoding processing described in known documents in the field of encoded imaging including Non-Patent Document 1.

[0037] In step H8, estimation of the depth of the object corresponding to each position of the decoded image is performed. That is, the depth estimation value of the object corresponding to each position in the decoded image obtained in step H7 is obtained based on the information obtained in step H7. Thereafter, a depth map of the subject may be generated based on the decoded image and the depth estimation value of the object corresponding to each position of the decoded image. Note that the derivation of the depth estimation value or the generation of the depth map is performed by, for example, the arithmetic control device 91.

[0038] <Examples of a plurality of masks> Here, examples of a plurality of masks prepared in advance will be described. Here, the incident region of light that arrives from the subject and enters the optical system (hereinafter also referred to as the light incident region) is a circular region having an almost perfect circular contour.

[0039] FIG. 3 is a diagram showing an example of a plurality of masks. The plurality of masks to be prepared are, for example, masks M1 to M4 shown in FIG. 3. The masks M1 to M4 shown in FIG. 3 are enlarged versions of the masks M1 to M4 shown in FIG. 1. Note that masks M1 to M4 show the mask patterns when viewed in the direction in which light from the subject is incident.

[0040] As shown in FIG. 3, masks M1 to M4 have a mode in which circular light shielding regions N1 to N4 for shielding light are formed at the upper right, upper left, lower left, and lower right, respectively, with respect to a shielding body that shields the light incident region of the optical system 81.

[0041] Mask M1 is a mask that has a pattern of shielding light only from a circular region inscribed in the upper right 1 / 4 region of the circular region that is the light incident region, and transmitting light in other regions. That is, mask M1 has a circular light shielding region N1 at the upper right of the light incident region.

[0042] Mask M2 is a mask that has a pattern of shielding light only from a circular region inscribed in the upper left 1 / 4 region of the circular region that is the light incident region, and transmitting light in other regions. That is, mask M2 has a circular light shielding region N2 at the upper left of the light incident region.

[0043] Mask M3 is a mask that has a pattern of shielding light only from a circular region inscribed in the lower left 1 / 4 region of the circular region that is the light incident region, and transmitting light in other regions. That is, mask M3 has a circular light shielding region N3 at the lower left of the light incident region.

[0044] Also, the mask M4 is a mask having a pattern that blocks light only in a circular region that is inscribed in the lower right 1 / 4 region of the circular region that is the light incident region, and transmits light in other regions. That is, the mask M4 has a circular light-shielding region N4 in the lower right of the light incident region.

[0045] 〈Correspondence between Mask Combinations and Edge Directions with High Depth Estimation Accuracy〉 FIG. 4 is a diagram showing the correspondence between mask combinations and edge directions with high depth estimation accuracy.

[0046] In the case of the combination of mask M1 and mask M2, the deviation direction of the light-shielding regions in the respective masks is the horizontal direction. Therefore, in the case of the combination of mask M1 and mask M2, the edge direction in which the blurring state hardly changes in the captured image with the mask is the horizontal direction. On the other hand, the edge direction in which the blurring state easily changes in the captured image with the mask is the vertical direction orthogonal to the horizontal direction which is the above deviation direction. That is, in the case of the combination of mask M1 and mask M2, the depth information of the subject is likely to appear, and the edge direction with high depth estimation accuracy is the vertical direction.

[0047] In the case of the combination of mask M1 and mask M3, the deviation direction of the light-shielding regions in the respective masks is the direction of 45 degrees diagonally to the right. The direction of 45 degrees diagonally to the right is the direction of a straight line obtained by rotating a vertical straight line 45 degrees clockwise. Therefore, in the case of the combination of mask M1 and mask M3, the edge direction in which the blurring state hardly changes in the captured image with the mask is the direction of 45 degrees diagonally to the right. On the other hand, the edge direction in which the blurring state easily changes in the captured image with the mask is the direction of 45 degrees diagonally to the left which is orthogonal to the above deviation direction of 45 degrees diagonally to the right. The direction of 45 degrees diagonally to the left is the direction of a straight line obtained by rotating a vertical straight line 45 degrees counterclockwise. That is, in the case of the combination of mask M1 and mask M3, the depth information of the subject is unlikely to appear, and the edge direction with high depth estimation accuracy is the direction of 45 degrees diagonally to the left.

[0048] By the same reasoning, in the case of the combination of mask M1 and mask M4, the edge direction with high depth estimation accuracy is the horizontal direction.

[0049] Also, in the case of the combination of mask M2 and mask M4, the edge direction with high depth estimation accuracy is the direction of 45 degrees diagonally to the right.

[0050] When summarizing the edge directions with high depth estimation accuracy for different combinations of masks, as shown in FIG. 4, the following correspondence can be obtained. (1) Combination of masks M1 and M2: vertical direction (2) Combination of masks M1 and M3: 45 degrees diagonally to the left (3) Combination of masks M1 and M4: horizontal direction (4) Combination of masks M2 and M4: 45 degrees diagonally to the right

[0051] 〈Example of method for determining representative edge direction〉 An example of a method for determining the representative edge direction in an image captured without a mask will be described. As methods for determining the representative edge direction, for example, there are the following methods.

[0052] 《First method for determining representative edge direction》 The first method is a method for determining the representative edge direction based on the edge image of a detected specific object.

[0053] FIG. 5 is a diagram for explaining the first method for determining the representative edge direction.

[0054] First, the setting of the object to be noted is performed in advance. When the depth estimation of the subject is used in the technical field of automotive driving support, the object to be noted is, for example, an automobile, a motorcycle, a bicycle, a wheelchair, a human, a dog, a utility pole, a traffic signal, etc.

[0055] Next, as shown in FIG. 5, in the imaging image P0 without a mask, the set object A1, which is the target object, is searched for. In FIG. 5, the imaging image P0 without a mask is an example of an image obtained by imaging the front from a moving automobile, and is an example when an automobile is detected as the set object A1. Note that, for the search of the set object A1, a detection method by template matching, a detection method by an AI (artificial intelligence), or the like is used.

[0056] Next, when the set object A1 is detected in the imaging image P0 without a mask, the representative edge direction is determined based on the edge image corresponding to the edge of the detected set object A1.

[0057] Specifically, for example, a plurality of partial image regions GB are set in the imaging image P0 without a mask. The plurality of partial image regions GB are set by dividing the entire image region of the imaging image P0 without a mask into a grid shape, that is, a matrix shape. The partial image region GB is, for example, an image region having a vertical × horizontal of 5 pixels × 5 pixels.

[0058] Next, as shown in FIG. 6, for each partial edge image region GE including the edge of the detected set object A1 among the set partial image regions GB, the edge direction E of the corresponding edge image is obtained. The edge direction E for each partial edge image region GE is obtained by selecting from among specific edge directions SE having high object depth estimation accuracy, which are associated with different combinations of masks. When obtaining the edge direction E of the partial edge image region GE, the specific edge direction SE having the smallest angular difference from the actual edge direction of the partial edge image region GE among the specific edge directions SE is selected.

[0059] The edge directions E obtained for each partial edge image BE are tabulated for each specific edge direction SE, and the specific edge direction SE with the largest number is determined as the representative edge direction DE.

[0060] 《Second method for determining the representative edge direction》 The second determination method is a method of determining a representative edge direction based on an edge image of an image region with little variation in shading or color.

[0061] FIG. 6 is a diagram for explaining a second method of determining a representative edge direction.

[0062] First, as shown in FIG. 6, in the non-mask captured image P0, flat regions T1, T2,... that are continuous regions where the degree of variation in shading or color is below the upper limit level and have an area equal to or greater than a threshold value are searched for. The flat regions T1, T2,... are, for example, continuous regions where the variance or standard deviation of pixel values is below a set upper limit value, and the area, i.e., the number of pixels, included in the region is equal to or greater than a set threshold value.

[0063] Next, when flat regions T1, T2,... are detected in the non-mask captured image P0, a representative edge direction DE is determined based on the edge image corresponding to the boundaries of the detected flat regions T1, T2,....

[0064] Specifically, for example, a plurality of partial image regions GB are set in the non-mask captured image P0. The plurality of partial image regions GB are set by dividing the entire image region of the non-mask captured image P0 into a matrix. The partial image region GB is, for example, an image region with a vertical × horizontal of 5 pixels × 5 pixels.

[0065] Then, as shown in FIG. 6, for each partial edge image region GE corresponding to the boundaries of the detected flat regions T1, T2,... among the set partial image regions GB, an edge direction E is obtained. When obtaining the edge direction E for each partial edge image region GE, from among the specific edge directions SE with high depth estimation accuracy of the object associated with different combinations of masks, the edge direction with the smallest angular difference from the edge direction in the actual partial edge image region GE is selected.

[0066] For each specific edge direction SE, the edge directions E of the partial edge image regions GE are aggregated, and the specific edge direction SE with the largest number is determined as the representative edge direction DE.

[0067] 《The Third Method for Determining the Representative Edge Direction》 The third method is a method for determining a representative edge direction based on edge images included in a plurality of partial image regions in an image taken without a mask.

[0068] FIG. 7 is a diagram for explaining the third method for determining the representative edge direction.

[0069] First, as shown in FIG. 7, a plurality of partial image regions GB are set in the image P0 taken without a mask. The plurality of partial image regions GB are set by dividing the entire image region of the image P0 taken without a mask into a matrix. The partial image region GB is, for example, an image region with a vertical × horizontal of 5 pixels × 5 pixels.

[0070] Next, for each of the plurality of partial image regions GB, for each specific edge direction SE associated with a different combination of masks and having a high depth estimation accuracy of the object, it is determined whether the edge of the specific edge direction SE is included. In this determination, if the deviation angle between the specific edge direction SE and the direction of the actual edge included in the partial image region GB is within a predetermined allowable error, it is determined that the edge of the specific edge direction SE is included in the partial image region GB.

[0071] Then, the number of edge directions of the edges determined to be included in the partial image region GB is aggregated for each specific edge direction SE. And the edge direction with the largest number is determined as the representative edge direction DE.

[0072] Note that the method for determining the representative edge direction DE is not limited to any of the above first to third methods. Further, the method for determining the representative edge direction DE may be a method combining two or more of the first to third methods. For example, priority may be given to two or more of the first to third methods, and the two or more methods may be implemented according to the priority until the representative edge direction DE is determined.

[0073] 《Regarding the Edge Image》 Here, the characteristics of the image that are preferably treated as the edge image in this embodiment will be described.

[0074] FIG. 8 is a diagram showing an example of a blurred edge image. Further, FIG. 9 is a diagram showing an example of a sharp edge image.

[0075] In this embodiment, it is preferable to treat an image representing a so-called blurred edge as the edge image. For example, an image Q1 in which the pixel value changes stepwise or gently with respect to the change in the position of the coordinates as shown in FIG. 8 is a blurred edge image. To give a more specific example, in a 256 - level grayscale image, an image in which five pixels are arranged in one direction and the pixel values change, for example, as 180, 150, 100, 50, 30 is a so-called blurred edge image. When decoding processing is performed on such a blurred edge image, the depth information of the object corresponding to the edge image can be extracted well, and high - precision depth estimation is possible.

[0076] On the other hand, in this embodiment, it is preferable not to treat an image representing a so-called sharp edge as an edge image. For example, an image Q2 in which pixel values change steeply with respect to the change in the position of coordinates as shown in FIG. 9 is a sharp edge image. To give a more specific example, in a 256 - level grayscale image, an image in which five pixels are arranged in one direction and the pixel values change, for example, as 180, 180, 30, 30, 30 is a so-called sharp edge image. Such a sharp edge image originally has no blur or almost no blur. Therefore, such a sharp edge image coincides with the shooting distance (distance between the subject and the optical system) calculated from the focal length of the optical system. However, in an optical system with a short focal length, the depth of field is deep, and the focus is achieved over a wide range, making it difficult to accurately estimate the depth.

[0077] As described above, in the subject depth estimation method according to Embodiment 1, first, in the unmasked captured image of the subject acquired in advance, a representative edge direction to be emphasized is determined. Then, as a combination of masks used for masked imaging of the subject necessary for estimating the depth of the subject in the captured image, from among a plurality of masks prepared in advance, a combination of masks with the relatively highest accuracy for estimating the depth of the object corresponding to the edge image in the representative edge direction is selected.

[0078] Therefore, according to the subject depth estimation method according to Embodiment 1, it becomes possible to estimate the depth of the corresponding object with higher stability and accuracy for the edge image in the representative edge direction emphasized in the captured image. Thus, according to the subject depth estimation method according to Embodiment 2, it becomes possible to provide a more practical DFD technology.

[0079] (Embodiment 2) The imaging device according to Embodiment 2 of the present application will be described. The imaging device according to Embodiment 2 of the present application includes an optical system into which light from a subject is incident, an imaging element that receives the light passing through the optical system, and a mask installation unit that creates a state in which any one of a plurality of pre-prepared masks is installed and a state in which no mask is installed with respect to the incident region of the light incident from the subject on the optical system, and a calculation control unit that outputs a signal for controlling the mask installation unit and the imaging element so that the subject is imaged, and acquires an image of the subject without a mask and an image of the subject with a mask. The calculation control unit includes a no-mask imaging process for obtaining a no-mask imaging image by imaging the subject without a mask, a determination process for determining a representative edge direction based on the edge image included in the no-mask imaging image, a selection process for selecting a combination of masks having the relatively highest depth estimation accuracy for an object corresponding to an image representing an edge in the same direction as the representative edge direction from among a plurality of pre-prepared masks, a with-mask imaging process for obtaining a plurality of with-mask imaging images by imaging the subject using the individual masks included in the selected mask combination, and a decoding process for obtaining information representing the depth at a plurality of positions of the subject by performing decoding based on the point spread function unique to the selected mask on the plurality of with-mask imaging images. The details of this imaging device are as follows.

[0080] <Example of the Configuration of the Imaging Device> FIG. 10 is a diagram showing an example of the configuration of the imaging device according to Embodiment 2. As shown in FIG. 10, the imaging device 1 according to Embodiment 1 includes an optical system unit 20, an imaging element 30, a liquid crystal mask unit 40, an optical system control unit 21, an imaging element control unit 31, a liquid crystal mask control unit 41, and a calculation control unit 10. Note that the "optical system unit 20" is an example of the "optical system" in the present application. The "liquid crystal mask unit 40" is an example of the "mask installation unit" in the present application.

[0081] The optical system unit 20 condenses the light L, which is the light emitted or reflected from the subject 3, and forms an image on the light-receiving surface 30a of the image sensor 30 described later. The optical system unit 20 includes a lens 20a. The lens 20a is, for example, a single-focus lens or a zoom lens. Although the lens 20a is generally a compound lens formed by combining a plurality of lenses, it may be a single lens. The optical system unit 20 may be an autofocus system or a fixed-focus system.

[0082] The image sensor 30 is an electronic component that performs photoelectric conversion. That is, the image sensor 30 forms an image of the light L, which is the light emitted or reflected from the subject 3, on the light-receiving surface 30a of the image sensor 30 through the optical system unit 20, photoelectrically converts the brightness or darkness of the image into the amount of electric charge, reads it out, and converts it into an electrical signal.

[0083] The image sensor 30 generally has a plurality of photoelectric conversion elements arranged in a two-dimensional array, and the light-receiving surface 30a is formed by these plurality of photoelectric conversion elements. The image sensor 30 is arranged at a position where the light L that has entered the optical system unit 20 from the subject 3 and passed through the optical system unit 20 is received by the light-receiving surface 30a. The image sensor 30 converts the intensity, that is, the brightness, of the light received by the light-receiving surface 30a into an electrical signal and outputs an image signal. The image sensor 30 may output a color image signal representing a color image or a monochrome image signal representing a monochrome image.

[0084] The image sensor 30 is, for example, a CCD (Charge Coupled Devices) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0085] The liquid crystal mask unit 40 is provided in front of the subject 3 side of the optical system unit 20. The liquid crystal mask unit 40 has a function of causing any one of a plurality of predetermined masks M to appear or not causing any mask M to appear. Note that the liquid crystal mask unit 40 may be provided inside the optical system unit 20.

[0086] In the present embodiment, the liquid crystal mask unit 40 is configured to be able to be in a state where any one of the masks M1 to M4 is installed or in a maskless state on the subject 3 side of the optical system unit 20.

[0087] The liquid crystal mask unit 40 has, for example, a liquid crystal light shutter 40a as shown in FIG. 10.

[0088] FIG. 11 is a diagram showing an example of the liquid crystal light shutter. As shown in FIG. 11, the liquid crystal light shutter 40a includes a light shielding portion BM and a plurality of segments R1 to R5.

[0089] The light shielding portion BM has an opening BMa through which the light L incident on the optical system unit 20 from the subject 3 passes. The light shielding portion BM is made of, for example, a black resin plate or a metal plate.

[0090] The segments R1 to R5 are arranged so as to divide the region of the opening BMa of the light shielding portion BM. The liquid crystal light shutter 40a has electrodes corresponding to each of the segments R1 to R5. Each of the segments R1 to R5 becomes either a light shielding state or a light transmitting state according to the voltage applied to the corresponding electrode.

[0091] The regions of the segments R1 to R4 among the segments R1 to R5 respectively correspond to the light shielding regions of the masks M1 to M4. The segment R5 corresponds to the region remaining after excluding the regions of the segments R1 to R4 from the region of the opening BMa of the light shielding portion BM.

[0092] By controlling the states of the segments R1 to R5, it becomes possible to achieve a state where the intended mask is installed or a state where no mask is installed.

[0093] For example, if segment R1 is in a light-shielding state and segments R2 to R5 are in a transmitting state, a state where mask M1 is installed is achieved. Alternatively, if segment R2 is in a light-shielding state and segments R1, R3 to R5 are in a transmitting state, a state where mask M2 is installed is achieved. Also, if segments R1 to R5 are in a light-transmitting state, a state where no mask is installed, that is, a maskless state, is achieved.

[0094] Note that the liquid crystal mask unit 40 may have a structure different from that of the liquid crystal light shutter. For example, the liquid crystal mask unit 40 may have a structure having a mechanism for mechanically switching a plurality of masks formed by a plate-like member. Also, for example, the liquid crystal mask unit 40 may have a structure having a plurality of diaphragm mechanisms that cover the entire light passing region of the light incident on the optical system unit 20 and have diaphragms capable of opening and closing at a plurality of different positions.

[0095] The optical system control unit 21 adjusts the position of the movable part included in the optical system unit 20 based on the control signal received from the arithmetic control unit 10. The optical system control unit 21 has, for example, a drive motor, and by operating the drive motor, at least a part of the lens is moved.

[0096] When the optical system unit 20 includes a zoom lens, the optical system control unit 21 may change the zoom magnification by moving a part of the lens group constituting the zoom lens, or may adjust the focus by moving the entire zoom lens. When the optical system unit 20 includes a single-focus lens, the focus may be adjusted by moving the entire lens. When the optical system unit 20 includes a diaphragm mechanism, the aperture diameter of the diaphragm may be adjusted by operating the diaphragm mechanism.

[0097] The imaging element control unit 31 executes imaging by reading an image signal output from the imaging element 30 based on a control signal received from the arithmetic control unit 10. The imaging element control unit 31 transmits the read image signal to the arithmetic control unit 10. Note that the shutter method for controlling the imaging element 30 to image the subject 3 may be, for example, a global shutter method or a rolling shutter method.

[0098] The liquid crystal mask control unit 41 controls the liquid crystal mask unit 40 based on a control signal received from the arithmetic control unit 10, and realizes a state in which the intended mask M is installed in the liquid crystal mask unit 40 or a state in which the mask M is not installed.

[0099] FIG. 12 is a diagram showing an example of the configuration of the arithmetic control unit 10 according to the second embodiment. As shown in FIG. 12, the arithmetic control unit 10 is, for example, a computer and includes a processor 11, a memory 12, and an interface 13.

[0100] The memory 12 stores a program P used for the processor 11 to execute various arithmetic processes, image processes, etc., and to execute various control processes. The memory 12 also stores temporarily or permanently the data processed by the processor 11.

[0101] The processor 11 executes various processes including arithmetic processes, image processes, and control processes by reading and executing the program P stored in the memory 12. When executing various processes, the processor 11 stores data in the memory 12 or accesses the data stored in the memory 12 to execute the processes.

[0102] In addition, as part of various processes, the processor 11 executes maskless imaging processing, representative edge direction determination processing, mask selection processing, first mask imaging processing, second mask imaging processing, decoding processing, subject depth estimation processing, depth map generation processing, data output processing, and imaging continuation determination processing. Details of these various processes will be described later.

[0103] The processor 11 transmits control signals to the optical system control unit 21, the image sensor control unit 31, and the liquid crystal mask control unit 41 in order to execute the above-mentioned maskless imaging process, representative edge direction determination process, mask selection process, first mask imaging process, and second mask imaging process.

[0104] The interface 13 is connected to the external device 2 and transmits the decoded image P3 or the depth map P4 generated in the arithmetic control unit 10 to the external device 2.

[0105] Note that all or part of the above computer may be configured by a semiconductor circuit such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a CPLD (Complex Programmable Logic Device).

[0106] The external device 2 is, for example, an image processing device, a vehicle driving support device, or the like. The image processing device performs processing such as blurring a background that is far from the optical system and emphasizing a subject to be focused on for a captured image. The vehicle driving support device detects, for example, the position or relative moving speed of an object around the vehicle, and issues a warning or controls the vehicle to avoid danger.

[0107] Note that an operation unit 17 and a display unit 18 are connected to the arithmetic control unit 10. The operation unit 17 is for receiving input operations from the user, and the display unit 18 is for visually outputting information to the user. The operation unit 17 is, for example, a keyboard, a mouse, a button, a dial, or the like. The display unit 18 is, for example, a liquid crystal panel, an organic EL panel, or the like. The operation unit 17 and the display unit 18 may be an integrated touch panel. The operation unit 17 and the display unit 18 may be provided on the external device 2 side.

[0108] <Flow of Operations in the Imaging Device> The flow of operations in the imaging device according to Embodiment 2 will be described. FIG. 13 is a flowchart showing an example of the flow of operations of the imaging device according to Embodiment 2. FIG. 14 is a diagram for explaining the flow from maskless imaging processing to mask selection processing.

[0109] As shown in FIG. 13, in step S1, maskless imaging processing is executed. That is, maskless imaging of the subject 3 is performed. Specifically, the arithmetic control unit 10 transmits control signals to the liquid crystal mask control unit 41, the optical system control unit 21, and the imaging element control unit 31 so that the subject 3 is imaged in a state where the mask M is not installed.

[0110] Based on the received control signal, the liquid crystal mask control unit 41 controls the liquid crystal mask unit 40 so that a state where the mask M is not installed is set. That is, the liquid crystal mask control unit 41 controls the liquid crystal mask unit 40 so that the segments R1 to R5 are in a light-transmitting state.

[0111] Next, based on the received control signal, the optical system control unit 21 controls the optical system unit 20 as needed so that the focal length and the like of the optical system unit 20 are in appropriate conditions.

[0112] Next, based on the received control signal, the imaging element control unit 31 controls the imaging element 30 so that the subject 3 is imaged, that is, so that the imaging image of the subject 3 represented by the output signal of the imaging element 30 is transmitted to the arithmetic control unit 10.

[0113] As described above, the arithmetic control unit 10 obtains a maskless imaging image P0 of the subject 3 as shown in FIG. 14 by performing maskless imaging of the subject 3.

[0114] In step S2, the representative edge direction determination process is executed. That is, the representative edge direction DE in the maskless captured image P0 is determined. Specifically, the arithmetic control unit 10 determines the representative edge direction DE in the maskless captured image P0 by using, for example, the method for determining the representative edge direction DE described above, based on the maskless captured image P0.

[0115] For example, using the third method for determining the representative edge direction described above, for each of the four predetermined specific edge directions SE, the number of edge directions that match or approximate the specific edge direction SE is counted, and the edge direction with the largest number is determined as the representative edge direction DE. In the example of FIG. 14, since the count value of the specific edge direction SE of 45 degrees to the right diagonal is the largest, this specific edge direction SE is determined as the representative edge direction DE.

[0116] In step S3, the mask selection process is executed. That is, the selection of the combination of masks M used for masked imaging is performed. Specifically, the arithmetic control unit 10 selects, from among the plurality of masks M1 to M4, the combination of masks M in which the specific edge direction SE with the relatively highest depth estimation accuracy of the subject matches the determined representative edge direction DE.

[0117] For example, when the representative edge direction DE is in the vertical direction, the combination of mask M1 and mask M2, or the combination of mask M3 and mask M4 is selected. When the representative edge direction DE is in the direction of 45 degrees to the left diagonal, the combination of mask M1 and mask M3 is selected. When the representative edge direction DE is in the horizontal direction, the combination of mask M1 and mask M4, or the combination of mask M2 and mask M3 is selected. When the representative edge direction DE is in the direction of 45 degrees to the right diagonal, the combination of mask M2 and mask M4 is selected.

[0118] In the example of FIG. 14, since the representative edge direction DE is at a 45-degree angle to the right diagonal, the combination of masks M in which the specific edge direction SE with the relatively highest depth estimation accuracy of the object is in the 45-degree direction to the right diagonal, that is, the combination of mask M2 and mask M4, is selected.

[0119] In step S4, masked imaging processing using the first mask is executed. That is, masked imaging of the subject 3 using the first mask among the selected combinations of masks M is performed. Specifically, the arithmetic control unit 10 transmits a control signal to the liquid crystal mask control unit 41 and the imaging element control unit 31 so that the subject 3 is imaged with the first mask M installed.

[0120] Based on the received control signal, the liquid crystal mask control unit 41 controls the liquid crystal mask unit 40 so that a state in which the first mask M is installed is set. That is, for each of the segments R1 to R5, the liquid crystal mask control unit 41 determines and sets whether to make it in a light transmission state or a light shielding state so that the segments R1 to R5 form the first mask M.

[0121] Next, based on the received control signal, the imaging element control unit 31 controls the imaging element 30 so that the subject 3 is imaged, that is, so that the imaging image of the subject 3 represented by the output signal of the imaging element 30 is transmitted to the arithmetic control unit 10.

[0122] As described above, the arithmetic control unit 10 acquires the first masked imaging image P1 of the subject 3 by performing imaging of the subject 3 using the first mask M.

[0123] In step S5, masked imaging processing using the second mask is executed. That is, masked imaging of the subject 3 using the second mask included in the selected combination of masks M is performed. Specifically, the arithmetic control unit 10 transmits a control signal to the liquid crystal mask control unit 41 and the imaging element control unit 31 so that the subject 3 is imaged with the second mask M installed.

[0124] Based on the received control signal, the liquid crystal mask control unit 41 controls the liquid crystal mask unit 40 so that the state in which the second mask M is installed is set. That is, the liquid crystal mask control unit 41 determines and sets, for each of the segments R1 to R5, whether to be in a light transmission state or a light shielding state so that the segments R1 to R5 form the second mask M.

[0125] Next, based on the received control signal, the imaging device control unit 31 controls the imaging device 30 so that the subject 3 is imaged, that is, so that the captured image of the subject 3 represented by the output signal of the imaging device 30 is transmitted to the arithmetic control unit 10.

[0126] As described above, the arithmetic control unit 10 obtains the captured image P2 of the subject 3 with the second mask by performing imaging of the subject 3 with the second mask M.

[0127] In step S6, a decoding process is executed. That is, a decoding process for the captured image with a mask is performed. Specifically, the arithmetic control unit 10 performs a decoding process on the captured image P1 with the first mask and the captured image P2 with the second mask based on the point spread function specific to the first mask M and the point spread function specific to the second mask M. When the decoding process is performed, a decoded image P3, which is an image with improved blur of the captured image P1 with the first mask or the captured image P2 with the second mask, and depth information capable of estimating the depth of the object corresponding to each position of the decoded image P3 are obtained.

[0128] In step S7, a subject depth estimation process is executed. That is, the depth of the object corresponding to each position of the decoded image P3 is estimated. Specifically, the arithmetic control unit 10 calculates an estimated depth value of the object corresponding to each position of the decoded image P3 based on the depth information obtained in step S6.

[0129] In step S8, depth map generation processing is executed. That is, a depth map P4 is generated. Specifically, the arithmetic control unit 10 generates a depth map P4 by associating a value representing the depth of the object corresponding to each position in the decoded image P3 with that position. Note that the depth map P4 represents, for a plurality of positions in the captured image, that is, each pixel or each partial image region, the depth information of the object represented by the pixel or the partial image region, in an associated manner.

[0130] In step S9, data output processing is executed. That is, the decoded image P3 and the depth map P4 are output. Specifically, the arithmetic control unit 10 outputs the decoded image P3 and the depth map P4 to the external device 2 via the interface 13.

[0131] In step S10, imaging continuation determination processing is executed. That is, it is determined whether to continue imaging. For example, when a signal requesting imaging stop is input due to an operation of the operation unit 17 by the user or a process executed by the external device 2, or when an error occurs from some source, the arithmetic control unit 10 determines to stop imaging. On the other hand, for example, when no signal requesting imaging stop is input or no error occurs, the arithmetic control unit 10 determines to continue imaging.

[0132] When it is determined to stop imaging, the arithmetic control unit 10 stops imaging and ends the process. On the other hand, when it is determined to continue imaging, the arithmetic control unit 10 returns the processing step to be executed to step S1 and continues the process.

[0133] As described above, in the imaging device 1 according to the second embodiment, first, in the maskless captured image of the subject acquired in advance, a representative edge direction to be emphasized is determined. Then, as a combination of masks used for capturing an image of the subject with a mask necessary for estimating the depth of the subject in the captured image, a combination of masks having the relatively highest accuracy in estimating the depth of the object corresponding to the edge image in the representative edge direction is selected from among a plurality of masks prepared in advance.

[0134] Therefore, according to the imaging device 1 according to Embodiment 2, for the edge image in the representative edge direction that is emphasized in the captured image, it becomes possible to estimate the depth of the corresponding object with higher stability and accuracy. Therefore, according to the imaging device according to Embodiment 1, it becomes possible to provide a more practical DFD technology.

[0135] Further, according to Embodiment 2, the state where the intended mask is installed or the state where the mask is not installed is realized by the liquid crystal light shutter. The liquid crystal light shutter can realize an arbitrary mask depending on the design of the segment, and there is no need to physically switch the hardware for switching the state of the mask. Therefore, the position of the mask can be accurately controlled, and the mechanism for switching the mask can also be simplified.

[0136] <Modification Example 1> Modification Example 1 will be described. In Embodiment 1 and Embodiment 2, the specific edge directions SE with high depth estimation accuracy corresponding to the combinations of masks are four types: the vertical direction, the left diagonal 45-degree direction, the horizontal direction, and the right diagonal 45-degree direction. However, the specific edge direction and the combination of masks corresponding to the specific edge direction may be five or more types.

[0137] FIG. 15 is a diagram showing an example of a plurality of types of combinations of masks according to Modification Example 1. The example of FIG. 15 is an example of a combination of masks prepared such that the specific edge direction SE with high object depth estimation accuracy is set at an angular interval of 22.5 degrees based on the vertical direction. That is, the prepared combination of masks has 16 types of combinations in which the specific edge direction SE is the vertical direction, the left diagonal 22.5-degree direction, the left diagonal 45-degree direction, the left diagonal 67.5-degree direction, ···, and the right diagonal 22.5-degree direction.

[0138] In Embodiment 2, the liquid crystal mask unit 40 is configured such that combinations of five or more types of masks described above can be formed. That is, the segmentation of segments in the liquid crystal light shutter 40a is designed so that these masks can be formed.

[0139] According to such Modification 1, there are more specific edge directions SE in which the depth estimation accuracy of an object is high. Therefore, the representative edge directions can be determined in directions with finer angles, and for the edge images of the representative edge directions that are regarded as important, the depth of the corresponding object can be estimated with higher accuracy.

[0140] <Modification 2> Modification 2 will be described. In Embodiment 1, Embodiment 2, and Modification 1, as the combination of masks used for imaging with masks, a combination of two masks is selected. However, as the combination of masks used for imaging with masks, a combination of three or more masks may be selected. Modification 2 is an example in which a combination of three or more masks is selected as the combination of masks used for imaging with masks.

[0141] FIG. 16 is a diagram showing an example of a combination of masks according to Modification 2. The example of FIG. 16 shows an example of a combination of three masks. For example, as shown in FIG. 16, a combination of three masks with the same misalignment direction of the light-shielding regions of the masks may be prepared as a selection candidate.

[0142] According to such Modification 2, as in Embodiment 1, for the edge images of the representative edge directions that are regarded as important in the captured image, the depth of the corresponding object can be estimated with higher accuracy.

[0143] <Modification 3> In Embodiment 2, the component that forms a mask in the imaging device is the liquid crystal mask unit 40. However, of course, it is not limited to this. For example, a plurality of types of mask plates, which are plate-like members with openings formed therein, may be prepared, and the mask plates may be mechanically switched in front of the subject side of the optical system unit 20 so that a desired mask is installed.

[0144] (Embodiment 3) A program according to Embodiment 3 of the present application will be described. The program according to Embodiment 3 of the present application causes a computer to perform a maskless imaging process of obtaining a maskless imaging image by imaging a subject without a mask, a determination process of determining a representative edge direction based on an edge image included in the maskless imaging image, a selection process of selecting, from among a plurality of pre-prepared masks, a combination of masks having relatively the highest depth estimation accuracy for an object corresponding to an image representing an edge in the same direction as the representative edge direction, a masked imaging process of obtaining a plurality of masked imaging images by imaging the subject using individual masks included in the selected mask combination, and a decoding process of obtaining information representing the depth at a plurality of positions of the subject by performing decoding based on a point spread function specific to the selected mask on the plurality of masked imaging images.

[0145] This program may be a program for causing a computer to execute the subject depth estimation method according to Embodiment 1. Further, this program may be a program for causing a computer to function as the arithmetic control unit 10 included in the imaging device according to Embodiment 2.

[0146] Note that a non-temporary tangible computer-readable recording medium on which the above program is recorded is also an embodiment of the present invention. By causing a computer to execute the above program, effects similar to those of the imaging device 1 according to Embodiment 2 can be obtained.

[0147] As described above, various embodiments of the present invention have been explained. However, the present invention is not limited to the above-described embodiments and includes various modifications. Also, the above-described embodiments have been explained in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations explained. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. All of these belong to the scope of the present invention. Furthermore, the numerical values and the like included in the text and the drawings are merely examples, and using different ones does not impair the effects of the present invention.

Explanation of Reference Numerals

[0148] 1... Imaging device, 2... External device, 3... Subject, 10... Arithmetic control unit, 11... Processor, 12... Memory, 13... Interface, 17... Operation unit, 18... Display unit, 20... Optical system unit, 21... Optical system control unit, 30... Image sensor, 31... Image sensor control unit, 40... Liquid crystal mask unit, 41... Liquid crystal mask control unit, 81... Optical system, 82... Image sensor, 90... Subject, 91... Arithmetic control device, D... Depth, L... Light, M, M1, M2, M3, M4... Mask, P... Program, P0... Imaging image without mask, P1... Imaging image with first mask, P2... Imaging image with second mask, P3... Decoded image, P4... Depth map

Claims

1. An optical system for receiving light from a subject, An image sensor for receiving the light passing through the optical system, A mask installation unit that creates a state in which any one of a plurality of pre-prepared masks is installed and a state in which none of the masks is installed with respect to an incident region of light incident from the subject on the optical system, An arithmetic control unit that outputs a signal for controlling the mask installation unit and the image sensor so that the subject is imaged, and acquires an image of the subject without a mask and an image of the subject with a mask, The arithmetic control unit, A maskless imaging process for obtaining a maskless imaging image by imaging the subject without the mask, A determination process for determining a representative edge direction based on an edge image included in the maskless imaging image, A selection process for selecting a combination of the masks in which the depth estimation accuracy of an object corresponding to an image representing an edge in the same direction as the representative edge direction is relatively the highest from among the plurality of pre-prepared masks, A masked imaging process for obtaining a plurality of masked imaging images by imaging the subject using each of the individual masks included in the selected combination of masks, A decoding process for obtaining information representing the depth at a plurality of positions of the subject by performing decoding based on a point spread function specific to the selected mask on the plurality of masked imaging images, An imaging device.

2. In the imaging device according to Claim 1, The mask has a light-shielding region in a part of the light incident region, The plurality of masks are two or more masks in which the positions of the light-shielding regions are different from each other, An imaging device.

3. In the imaging device according to Claim 2, The combination of the masks in which the depth estimation accuracy of an object corresponding to an image representing an edge in the same direction as the representative edge direction is relatively the highest are two or more masks in which the deviation directions of their respective main apertures are orthogonal to the representative edge direction, An imaging device.

4. In the imaging device according to any one of Claims 1 to 3, The determination process is a process of detecting a predetermined object in the maskless imaging image and determining the representative edge direction based on an edge image corresponding to the boundary of the detected object, An imaging device.

5. In the imaging device according to any one of claims 1 to 3, the determination process is a process of detecting, in the maskless imaging image, a continuous region where the degree of variation in shade or color is equal to or less than the upper limit level and having an area equal to or greater than a threshold value, and determining the representative edge direction based on an edge image corresponding to the boundary of the detected region. Imaging device.

6. In the imaging device according to any one of claims 1 to 3, the determination process is a process of setting a plurality of partial image regions in the maskless imaging image and determining the representative edge direction based on the edge images included in the partial image regions. Imaging device.

7. In the imaging device according to any one of claims 1 to 3, the mask installation unit has a liquid crystal optical shutter. Imaging device.

8. An image without a mask is obtained by imaging a subject without a mask, a representative edge direction is determined based on the edge image included in the image without a mask, from among a plurality of pre-prepared masks, a combination of masks having the relatively highest depth estimation accuracy for an object corresponding to an image representing an edge in the same direction as the representative edge direction is selected, a plurality of images with masks are obtained by imaging the subject using the individual masks included in the selected combination of masks, information representing the depth at a plurality of positions of the subject is obtained by performing decoding based on a point spread function specific to the selected mask on the plurality of images with masks. Subject depth estimation method.

9. In the subject depth estimation method according to claim 8, the mask has a light-shielding region in a part of the incident region of light incident on the optical system used for imaging the subject from the subject, the plurality of masks are two or more masks having different positions of the light-shielding regions from each other. Subject depth estimation method.

10. In the subject depth estimation method according to claim 9, the combination of masks having the relatively highest depth estimation accuracy for an object corresponding to an image representing an edge in the same direction as the representative edge direction are two or more masks in which the shifted directions of the respective light-shielding regions are orthogonal to the representative edge direction. Subject depth estimation method.

11. In the subject depth estimation method according to any one of claims 8 to 10, detect a predetermined object in the unmasked captured image, and determine the representative edge direction based on an edge image corresponding to the boundary of the detected object. Subject depth estimation method.

12. In the subject depth estimation method according to any one of claims 8 to 10, in the unmasked captured image, detect a continuous region where the degree of variation in shading or color is below an upper limit level and has an area equal to or greater than a threshold value, and determine the representative edge direction based on an edge image corresponding to the boundary of the detected region. Subject depth estimation method.

13. In the subject depth estimation method according to any one of claims 8 to 10, in the unmasked captured image, set a plurality of partial image regions, and determine the representative edge direction based on the edge images included in the partial image regions. Subject depth estimation method.

14. A program for causing a computer to perform an unmasked imaging process of obtaining an unmasked captured image by imaging a subject without a mask, perform a determination process of determining a representative edge direction based on an edge image included in the unmasked captured image, perform a selection process of selecting a combination of the masks that has the relatively highest accuracy in estimating the depth of an object represented by an image showing an edge in the same direction as the representative edge direction from among a plurality of the masks prepared in advance, perform a masked imaging process of obtaining a plurality of masked captured images by imaging the subject using the individual masks included in the selected combination of the masks, and perform a decoding process of obtaining information representing the depth at a plurality of positions of the subject by performing decoding based on a point spread function specific to the selected mask on the plurality of masked captured images.

Images