Depth map generation system, generation method, and program

Abstract

To provide a system that can expand a depth map use scene.SOLUTION: A generation system 10 includes a liquid crystal shutter 14 that forms a coding opening part 14a, and a depth calculation unit 11c that calculates the depth for each unit region in a depth map using a capture image f acquired in an imaging element 13. The coding opening part 14a includes an opening pattern B including a transmission region R1 with a first light transmittance T1, a half-transmission region R2 with a second light transmittance T2, which is lower than the first light transmittance T1, and a blocking region R3 with a third light transmittance T3, which is lower than the second light transmittance T2.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a system, a method, and a program for generating a depth map. [Background technology]

[0002] Non-Patent Document 1 proposes a technology for generating a depth map from an image captured through a coded aperture. In this document, two coded apertures with different aperture patterns (shapes of light-transmitting and blocking regions) are used. The coded aperture can be formed by a liquid crystal shutter. By using two coded apertures in combination, it is possible to prevent the occurrence of frequency bands with a power spectrum of 0 in the filter (filter for generating a restored image) used in the process of generating the depth map. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] C.Zhou, S.Lin, and S.Nayar: Coded Aperture Pairs for Depth from Defocus, IEEE international conference on computer vision, 2009 Summary of the Invention [Problem to be solved by the invention]

[0004] However, as proposed in Non-Patent Document 1, the method of using two coded apertures requires that the subject be photographed twice using each of the two coded apertures, and the subject must not move at all during that time, so that the situations in which this method can be used are limited. Furthermore, even if the operating speed of the liquid crystal shutter is increased and the exposure time is shortened, it is difficult to use this method for capturing moving images, for example. [Means for solving the problem]

[0005] The depth map generation system proposed in the present disclosure includes a liquid crystal shutter that forms a coding aperture, an image sensor that captures light transmitted through the coding aperture, and a depth calculation unit that calculates a depth for each unit area in a depth map using a captured image acquired by the image sensor. The opening pattern of the coding aperture includes a first region having a first light transmittance, a second region having a second light transmittance lower than the first light transmittance, and a third region having a third light transmittance lower than the second light transmittance.

[0006] The method for generating a depth map proposed in the present disclosure includes the steps of forming a coded aperture using a liquid crystal shutter, capturing light passing through the coded aperture, and calculating a depth for each unit area in the depth map using a captured image acquired by the image sensor. In the step of forming the coded aperture, an aperture pattern is formed that includes a first region having a first light transmittance, a second region having a second light transmittance lower than the first light transmittance, and a third region having a third light transmittance lower than the second light transmittance.

[0007] The program proposed in the present disclosure is a program that causes a computer to function as a means for forming a coded aperture using a liquid crystal shutter, a means for capturing light passing through the coded aperture, and a means for calculating a depth for each unit area in a depth map using a captured image acquired by the imaging element. The means for forming the coded aperture forms an aperture pattern including a first region having a first light transmittance, a second region having a second light transmittance lower than the first light transmittance, and a third region having a third light transmittance lower than the second light transmittance.

[0008] According to the depth map generation system, depth map generation method, and program proposed in the present disclosure, the number of times a subject needs to be photographed to generate a depth map can be reduced to, for example, one time, thereby expanding the scope of use of these generation systems, etc. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a cross-sectional view showing an example of an imaging system of a depth map generating system. [Diagram 2] FIG. 2 is a block diagram showing hardware of a depth map generation system. [Diagram 3] 11 is a block diagram showing the functions of a control unit of the depth map generating system. FIG. [Figure 4] 13A and 13B are diagrams illustrating examples of aperture patterns of coded apertures. [Diagram 5] FIG. 5 is a diagram for explaining the significance of the opening pattern shown in FIG. [Figure 6] FIG. 4 is a flow chart illustrating an example of a process executed by a control unit. [Figure 7] 11A and 11B are diagrams for explaining a calculation process of an estimated depth. [Figure 8] 11 is a diagram for explaining processing by a depth calculation unit. FIG. [Figure 9] FIG. 5 is a diagram for explaining the significance of the aperture pattern shown in FIG. 4, showing an aperture pattern having a transmission region of radius a at its center. [Figure 10] 13A and 13B are diagrams showing modified examples of opening patterns. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The depth map generating system, generating method, and program proposed in this disclosure will be described below.

[0011] [Generation system hardware] 1 is a cross-sectional view showing an imaging system N of a generation system 10, which is an example of a depth map generation system proposed in this disclosure. As shown in the figure, the generation system 10 has a liquid crystal shutter 14 and an image sensor 13.

[0012] The imaging element 13 is an image sensor such as a Complementary Metal Oxide Semiconductor (CMOS) sensor or a Charge Coupled Device (CCD).

[0013] The liquid crystal shutter 14 includes a plurality of pixels. The liquid crystal shutter 14 has a coded aperture 14a in a portion thereof. The control unit 11, which will be described later, drives the liquid crystal of the coded aperture 14a so as to form a predefined aperture pattern. The aperture pattern will be described in detail later.

[0014] 1, the generation system 10 may include a lens 15 disposed between the coded aperture 14a and the imaging element 13. Light that passes through the coded aperture 14a and the lens 15 enters the imaging element 13.

[0015] Fig. 2 is a block diagram showing hardware of the generation system 10. The generation system 10 has a control unit 11, a storage unit 12, and an input unit 16 in addition to the liquid crystal shutter 14, the image sensor 13, and the lens 15 shown in Fig. 1.

[0016] The control unit 11 includes at least one processor, such as a CPU (Central Processing Unit) or a GPU (Graphical Processing Unit). Image data acquired by the imaging element 13 is provided to the control unit 11. The control unit 11 uses this image data to generate a depth map indicating the distance to the subject.

[0017] The storage unit 12 includes a main storage unit and an auxiliary storage unit. For example, the main storage unit is a volatile memory such as a RAM (Random Access Memory), and the auxiliary storage unit is a non-volatile memory such as a ROM (Read Only Memory), an EEPROM (Electrically Erasable and Programmable Read Only Memory), a flash memory, or a hard disk. The control unit 11 executes a program stored in the storage unit 12 to control the liquid crystal shutter 14 and calculate the depth (distance to the subject). The processing executed by the control unit 11 will be described later.

[0018] The generation system 10 may be, for example, a mobile device such as a smartphone or a tablet PC (Personal Computer), or a personal computer connected to a camera.

[0019] The input unit 16 may be a touch sensor attached to the liquid crystal shutter 14. The input unit 16 may also be a pointing device such as a keyboard or a mouse. The input unit 16 inputs a signal to the control unit 11 in response to a user's operation.

[0020] [Generation system features] 3 is a block diagram showing functions of the control unit 11. The control unit 11 has, as its functions, an image acquisition unit 11a and a depth calculation unit 11c. The image acquisition unit 11a also includes an aperture control unit 11b. These functions are realized by the control unit 11 operating in accordance with a program stored in the storage unit 12.

[0021] The aperture control unit 11b controls the liquid crystal shutter 14 to form a predetermined aperture pattern, thereby forming the coding aperture 14a. The image acquisition unit 11a controls the image sensor 13 and the coding aperture 14a, and captures an image f through the coding aperture 14a. (Hereinafter, this image f will be referred to as a captured image.)

[0022] [Opening pattern] FIG. 4 is a diagram showing an example of an aperture pattern B formed by the coded aperture 14a. As shown in the figure, the aperture pattern B has a transmissive region R1 that transmits light and a shielding region R3 that shields light. The aperture pattern B also has a semi-transmissive region R2 in addition to the transmissive region R1 and the shielding region R3. The transmissive region R1, the semi-transmissive region R2, and the shielding region R3 have light transmittances T1, T2, and T3, respectively. The light transmittance T1 is, for example, the upper limit of the range of light transmittance adjustment by the liquid crystal shutter 14, and the light transmittance T3 is, for example, the lower limit of the range of light transmittance adjustment by the liquid crystal shutter 14. That is, the light transmittance T1 is, for example, the upper limit (100%) of the light transmittance that the aperture control unit 11b can achieve by the liquid crystal shutter 14, and the light transmittance T3 is the lower limit (0%) of the light transmittance that the aperture control unit 11b can achieve by the liquid crystal shutter 14. The light transmittance T2 is a transmittance between the light transmittances T1 and T3. That is, these light transmittances have the relationship "T1>T2>3". When the light transmittances T1 and T3 are 100% and 0%, respectively, the light transmittance T2 may be, for example, 40% to 60%. More specifically, the light transmittance T2 may be substantially the intermediate value (50%) between the light transmittances T1 and T3.

[0023] Note that light transmittance T1 (100%) does not necessarily mean that all of the light incident on the transmissive region R1 is transmitted through the transmissive region R1. A portion of the light incident on the transmissive region R1 may be absorbed by or reflected by the liquid crystal shutter 14. Also, light transmittance T3 (0%) does not necessarily mean that all of the light incident on the shielding region R3 is not transmitted through the shielding region R3. A portion of the light incident on the shielding region R3 may be transmitted through the liquid crystal shutter 14.

[0024] The transmissive region R1 and the semi-transmissive region R2 may each be, for example, circular or annular with the same center. In the example shown in Fig. 4, the semi-transmissive region R2 is an area having a larger outer diameter than the transmissive region R1. That is, the transmissive region R1 is located inside the semi-transmissive region R2. The semi-transmissive region R2 is annular. The shielding region R3 is formed outside the semi-transmissive region R2.

[0025] The shapes of the transmissive region R1, the semi-transmissive region R2, and the shielding region R3 are not limited to the example shown in Fig. 4. For example, the transmissive region R1 may be formed in an annular shape, and the semi-transmissive region R2 may be formed inside the transmissive region R1.

[0026] The point spread function (PSF) corresponding to the aperture pattern B having three regions R1, R2, and R3 with different transmittances is substantially equal to the sum of two PSFs corresponding to the two different aperture patterns. With reference to Fig. 5, the PSF corresponding to the aperture pattern B illustrated in Fig. 4 is substantially equal to the sum of PSFs corresponding to the two different aperture patterns B1 and B2. The aperture pattern B1 illustrated in Fig. 5 has a shielding region R3' having the same size and shape as the shielding region R3 illustrated in Fig. 4, and a semi-transmitting region R1' (region with a light transmittance of 50%) having the same size and shape as the transmitting region R1 illustrated in Fig. 4. Furthermore, aperture pattern B2 shown in FIG. 5 has a shielding region R3" and a semi-transparent region R2' (region with a light transmittance of 50%) having the same size as the semi-transparent region R2 shown in FIG. 4. By using a coded aperture 14a having such aperture pattern B, it is possible to calculate a depth that is substantially the same as a depth calculated from two images obtained through two coded apertures having aperture patterns B1 and B2, respectively. Furthermore, it is sufficient to capture an image through coded aperture 14a only once. As a result, the range of applications of generation system 10 can be expanded compared to a system using two coded apertures. The significance of aperture pattern B having semi-transparent region R2 will be explained in detail later.

[0027] [Calculating the estimated depth] The depth calculation unit 11c generates a depth map from the capture image f obtained by the image acquisition unit 11a. The depth calculation unit 11c calculates the depth of the capture image f (the distance from the imaging system N to the subject) for each of the multiple unit areas constituting the depth map. The unit area of ​​the depth map may be, for example, each pixel in the capture image f, or may be an area including multiple pixels in the capture image f. That is, the unit area of ​​the depth map may be, for example, an area including multiple adjacent pixels (for example, 2×2, 3×3, etc.).

[0028] Fig. 6 is a flow diagram showing an example of the process of the depth calculation unit 11c. The process of the depth calculation unit 11c shown in Fig. 6 is started after the capture image f is acquired by the image acquisition unit 11a.

[0029] First, the depth calculation unit 11c performs a two-dimensional Fourier transform on the captured image f (S101). In the following, the frequency characteristic of the captured image f obtained by the two-dimensional Fourier transform is represented as F.

[0030] In the generating system 10, a plurality of point spread functions (PSFs) corresponding to a plurality of discretely set reference depths are prepared. The reference depths are candidate values ​​for the distance to the subject, such as 100 mm, 200 mm, and 300 mm. Each PSF has a shape corresponding to the aperture pattern B of the coding aperture 14a. Also, each PSF has a size according to the corresponding reference depth. As the reference depth increases, the size of the PSF decreases. The plurality of PSFs are stored in the storage unit 12 as frequency characteristics obtained by performing a two-dimensional Fourier transform on them. The depth calculation unit 11c acquires this PSF as a frequency characteristic from the storage unit 12. This frequency characteristic is also called an optical transfer function (OTF).

[0031] The depth calculation unit 11c uses the capture image f and the PSF to generate a restored image corresponding to each of a plurality of reference depths (S102). Hereinafter, this restored image is referred to as a "reference depth restored image." Specifically, the depth calculation unit 11c uses the following equation 1 to calculate the frequency characteristics of the reference depth restored image corresponding to the frequency characteristics F of the capture image f and the frequency characteristics of the PSF.

[0032]

number

[0033] In the number 1 above, each letter represents the following element: F n : Reference depth-reconstructed image in frequency domain F: Frequency characteristic of captured image f obtained by aperture pattern B ·K n : Frequency characteristic of PSF (optical transfer function) corresponding to the nth size having a shape corresponding to aperture pattern B η: Noise term when the noise occurrence probability at each pixel is assumed to follow a Gaussian distribution (for example, η = 0.005) α: The variance of the first derivative of the pixel values ​​of each pixel, set to match the first derivative of the pixel values ​​in a natural image (e.g., α=250) ·Σ n |G n |: The sum of the absolute value of the frequency characteristic of the first-order differential filter corresponding to the X direction (horizontal direction of the captured image) and the absolute value of the frequency characteristic of the first-order differential filter corresponding to the Y direction (vertical direction of the captured image) · K with a line through it n : Frequency characteristic of PSF K n Complex conjugate of This formula (1) is derived from formula (11) presented in the following paper. Anat Levin, Rob Fergus, Fredo Durand, Bill Freeman, Image and Depth from a Conventional Camera with a Coded Aperture, ACM Transactions on Graphics, Vol. 26, No. 3, Article 70, Publication date: July 2007.

[0034] The size of the PSF becomes smaller as the distance (depth) from the imaging system N to the subject increases. Therefore, the frequency characteristic K n is also determined according to the distance from the imaging system N to the subject. n The subscript “n” in the above is a natural number such as 1, 2, 3, etc., which corresponds to a reference depth D of 100 mm, 200 mm, 300 mm, etc. For example, the frequency characteristic K 1 is the frequency characteristic of the PSF for a subject 100 mm away from the imaging system N, and frequency characteristic K 2 is the frequency characteristic of the PSF for an object that is 200 mm away from the imaging system N. The number of reference depths may be determined according to the resolution of the depth map, for example, 10, 20, 30, etc.

[0035] In S102, the depth calculation unit 11c calculates the frequency characteristic F of the captured image f and the frequency characteristic K of the PSF using Equation 1. n The frequency characteristic F of the reference depth restored image according to n That is, the depth calculation unit 11c calculates the frequency characteristics K 1 , K 2 , K 3 , and the frequency characteristic F of the captured image f are used to obtain the frequency characteristic F of multiple reference depth restored images. n For example, the depth calculation unit 11c calculates the frequency characteristic K 1 Based on the frequency characteristic F of the captured image f, the frequency characteristic F of the reference depth restored image is calculated. 1 Calculate the frequency characteristic K2 Based on the frequency characteristic F of the captured image f, the frequency characteristic F of the reference depth restored image is calculated. 2 Calculate.

[0036] By using Equation 1, an object whose distance from the imaging system N is equal to the reference depth appears clearly in the reference depth restored image. For example, an object located 100 mm from the imaging system N appears clearly in the reference depth restored image. 1 ), ringing appears clearly in the reference depth restored image obtained by the PSF. On the other hand, ringing appears for the same subject in the reference depth restored image obtained by the PSF corresponding to other reference depths such as 200 mm and 300 mm. Therefore, the depth calculation unit 11c calculates and searches for a reference depth at which the occurrence of ringing is minimal for each unit area of ​​the depth map. Hereinafter, the reference depth calculated in this manner is referred to as an estimated depth.

[0037] The depth calculation unit 11c calculates the frequency characteristic F n is inverse Fourier transformed to obtain the reference depth restored image f n Then, the depth calculation unit 11c generates a reference depth restored image f n and the captured image f, a value (deviation value) according to the degree of deviation between the reference depth and the actual depth is calculated for each pixel (S103). The depth calculation unit 11c calculates this deviation value, for example, by using the following equation 2.

[0038]

number

[0039] In number 2, each letter represents the following element: e n : Deviation value f: Capture image ·k n : a point spread function (PSF) corresponding to aperture pattern B and having the nth size f n: The reference depth restored image F obtained using Equation 1 n Inverse Fourier transform of ·k n* f n :k n and f n Convolution of

[0040] If the nth reference depth and the actual depth match, k n* f n The pixel values ​​of the image obtained by the above are substantially the same as the pixel values ​​of the captured image f, and the deviation value e n is effectively 0.

[0041] The depth calculation unit 11c calculates a window region W i The deviation value is calculated for this window region W and used as the energy function (S104). i may correspond to a unit area of ​​the depth map. For example, as shown in FIG. 7, the depth calculation unit 11c calculates a window area W i A plurality of window regions W are defined in the depth map. This calculation by the depth calculation unit 11c is expressed by, for example, the following equation 3. In equation 3, the window region W i The energy function E n_i is the window area W i The deviation value e calculated for multiple pixels (j) included in n It may be the average value of the square of (j).

[0042]

number

[0043] This energy function E nis calculated for each of a plurality of reference depths set in advance. The depth calculation unit 11c calculates the energy function E n The depth calculation unit 11c calculates the reference depth that minimizes the depth of each window region W as the estimated depth (S105). i Calculate the estimated depth for

[0044] In S105, the energy function E n When evaluating the energy function E n The size (magnification) of the PSF corresponding to the weight λ may be taken into consideration. n ×Energy function E n_i " is referred to as the evaluation value.

[0045]

number

[0046] 8 is a diagram for explaining this process by the depth calculation unit 11c. In FIG. 8, the horizontal axis represents the reference depth, and the vertical axis represents the depth of a certain window region W i The evaluation value calculated for n ×Energy function E n_i ) The evaluation value is smallest at a reference depth that is equal to or close to the actual depth. In the example of FIG. 8, the reference depth: 0_i The evaluation value is smallest at a position (e.g., 1000 mm). Therefore, the subject appearing in this window region is estimated to be at a position of 1000 mm from the imaging system N. In S105, the depth calculation unit 11c searches for a reference depth that provides the smallest evaluation value, and refers to the obtained reference depth as the estimated depth D. 0_iLet us assume that.

[0047] The calculation process of the depth map is not limited to the example described here. For example, the depth calculation unit 11c calculates an evaluation value (weight λ n ×Energy function E n_i ) and the reference depth, and the reference depth at which the function has a minimum is called the estimated depth D. 0_i It may be calculated as:

[0048] The depth calculation unit 11c determines whether or not the estimated depth has been calculated for all window regions W (all unit regions of the depth map) (S106). If there is a window region W for which the estimated depth has not yet been calculated, the depth calculation unit 11c returns to S104 and executes the subsequent processes. When the estimated depth has been calculated for all window regions W, the process of the depth calculation unit 11c ends. This results in a depth map.

[0049] [Significance of aperture patterns with semi-transparent regions] 5, the aperture pattern B of the coded aperture 14a having the semi-transmissive region R2 can perform the same function as the two aperture patterns B1 and B2, as will be described below.

[0050] First, a general aperture pattern that does not have a semi-transparent region will be described. The aperture pattern B3 shown in Fig. 9 has a transparent region R1 with a radius of a in the center. This aperture pattern B3 is expressed by the following equation 5.

[0051]

number

[0052] In the formula 5, h=1 indicates a light transmitting region R1, and h=0 indicates a light blocking region R3. The two-dimensional Fourier transform of the aperture pattern B3 expressed by the formula 5 is expressed by the following formula 6 in the polar coordinate system.

[0053]

number

[0054] This number 6 can be expressed as the following number 7 using the Bessel function. In number 7, J 1 represents the Bessel function of the first kind with n=1.

[0055]

number

[0056] Number 7 can be transformed into the following number 8.

number

[0057] Next, the aperture pattern B shown in Fig. 4 will be described. The aperture pattern B shown in Fig. 4 can be expressed by the following formula 9. As described above, in aperture pattern B, region R1 is a transmissive region, region R2 is a semi-transmissive region, and region R3 is a shielding region.

[0058]

number

[0059] Similar to Equation 6, the two-dimensional Fourier transform of the aperture pattern B expressed by Equation 9 can be expressed in a polar coordinate system as Equation 10 below.

[0060]

number

[0061] In Equation 10, the first term on the right hand side represents the transmissive region R1, and the second term on the right hand side represents the semi-transmissive region R2. Equation 10 can be transformed into the following Equation 11.

[0062]

number

[0063] In this formula 11, the first term on the right side corresponds to the two-dimensional Fourier transform of an aperture with a light transmittance of 50% and a radius a (i.e., aperture pattern B1 in FIG. 5), and the second term on the right side corresponds to the two-dimensional Fourier transform of an aperture pattern with a light transmittance of 50% and a radius b (i.e., aperture pattern B2 in FIG. 5). In other words, this formula 11 indicates that the two-dimensional Fourier transform of aperture pattern B shown in FIG. 4 is equal to the sum of the two-dimensional Fourier transform of the semi-transparent region R1' (see FIG. 5) with a radius a and the two-dimensional Fourier transform of the semi-transparent region R2' (see FIG. 5) with a radius b. Therefore, the coded aperture 14a having the aperture pattern B shown in FIG. 4 can function as two coded apertures each including semi-transparent regions R1' and R2'. Therefore, it is sufficient to capture an image through the coded aperture 14a only once.

[0064] [PSF of blurred image created by aperture pattern with semi-transparent area] The point spread function (PSF) of the blurred image obtained by the coded aperture is a figure similar to the aperture pattern of the coded aperture. It is known that the PSF of the blurred image is expressed by the following equation 12 using a function representing the aperture pattern and a magnification m.

[0065]

number

[0066] Therefore, the PSF of the blurred image obtained by aperture pattern B3 (see FIG. 9) is expressed by the following Equation 13 using Equation 12 and Equation 8 which represents aperture pattern B3.

[0067]

number

[0068] Moreover, the PSF of a blurred image obtained by the aperture pattern B shown in FIG. 4 is expressed by the following equation 14 using equations 11, 12, and 13.

[0069]

number

[0070] As described above, the two-dimensional Fourier transform of the PSF of the blurred image obtained by aperture pattern B shown in Figure 4 is identical to the sum of the two-dimensional Fourier transform of the PSF of the blurred image obtained by semi-transparent region R1' of radius a (see Figure 5) and the two-dimensional Fourier transform of the PSF of the blurred image obtained by semi-transparent region R2' of radius b (see Figure 5).

[0071] [Variations] Note that the depth map generation system described in the present disclosure is not limited to the generation system 10 described above, and various modifications may be made.

[0072] For example, the aperture pattern of the encoding aperture 14a is not limited to the aperture pattern B shown in FIG. 4. The aperture pattern of the encoding aperture may have a plurality of transmissive regions R1 and / or a plurality of semi-transmissive regions R2.

[0073] FIG. 10 is a diagram showing a modified example of the aperture pattern. In this figure, the aperture pattern B4 has transmissive regions R21a and R21b, semi-transmissive regions R22a and R22b, and shielding regions R23a and R23b. The two semi-transmissive regions R22a and R22b are regions obtained by removing the overlapping portion from two circular regions with a transmittance of 50% having an overlapping portion. Also, the overlapping portion of these two circles is the shielding region R23a. Further, the transmissive regions R21a and R21b are regions obtained by removing the two semi-transmissive regions R22a and R22b and the shielding region R23a from a circular region with a transmittance of 100%.

[0074] As yet another example, in the aperture pattern of the encoding aperture 14a, the light transmittance was set in three steps (T1, T2, T3). However, in the aperture pattern of the encoding aperture 14a, the light transmittance may be set in four steps or five steps. For example, in the aperture pattern of the encoding aperture 14a, the transmissive region R1 (light transmittance T1), the semi-transmissive region R2 (light transmittance T2 < T1), and the semi-transmissive region R2' (light transmittance T2' < T2) may have a shielding region R3 (light transmittance T3 < T2').

[0075] Also, in the example described with reference to FIG. 6, a window region W including a plurality of pixels was defined, and the estimated depth was calculated assuming that the depth was uniform within this window region W. However, the method for calculating the estimated depth is not limited to this. For example, the estimated depth may be calculated for each pixel of the captured image f.

[0076] [Summary] (1) As described above, the depth map generation system 10 proposed in the present disclosure includes a liquid crystal shutter 14 that forms a coding aperture 14a, an image sensor 13 that captures light transmitted through the coding aperture 14a, and a depth calculation unit 11c that calculates a depth for each unit region (window region W) in the depth map using a captured image f acquired by the image sensor 13. The coding aperture 14a has aperture patterns B·B4 that include a first region (transmissive region) R1 having a first light transmittance T1, a second region (semi-transmissive region) R2 having a second light transmittance T2 lower than the first light transmittance T1, and a third region (shielding region) R3 having a third light transmittance T3 lower than the second light transmittance T2.

[0077] According to this generation system 10, the number of times an object needs to be photographed to generate a depth map can be reduced to, for example, one time, which expands the range of applications for these generation systems and the like.

[0078] (2)In the generation system 10 of (1), the first light transmittance T1 is an upper limit of the light transmittance that can be realized by the liquid crystal shutter 14, the third light transmittance T3 is a lower limit of the light transmittance that can be realized by the liquid crystal shutter 14, and the second light transmittance T2 is a light transmittance between the first light transmittance and the third light transmittance.

[0079] (3) In the generating system 10 of (1) or (2), when the first light transmittance is 100% and the third light transmittance is 0%, the second light transmittance is 40% to 60%.

[0080] (4) In any one of the production systems 10 of (1) to (3), when the first light transmittance is 100% and the third light transmittance is 0%, the second light transmittance is substantially 50%.

[0081] (5) In any one of the generating systems 10 described in (1) to (4), the first region (transmissive region) R1 and the second region (semi-transmissive region) R2 are circular and have the same center.

[0082] (6) In the generating system 10 of any one of (1) to (5), the aperture pattern B4 of the coded aperture 14a has a plurality of first regions (transmissive regions) R1.

[0083] (7) In the generating system 10 of any one of (1) to (6), the aperture pattern B4 of the coded aperture 14a has a plurality of second regions (semi-transparent regions) R2.

[0084] (8) In any one of the generating systems 10 described in (1) to (7), the depth calculation unit 11c acquires a plurality of PSFs corresponding to a plurality of reference depths that are discretely set and corresponding to the opening patterns B and B4, and calculates a deviation value (E n ) is calculated for each unit area of ​​the depth map, and the deviation value (E n ) the depth is calculated for each unit region of the depth map.

[0085] (9) The method for generating a depth map proposed in the present disclosure includes the steps of forming a coded aperture 14a using a liquid crystal shutter 14, capturing light passing through the coded aperture 14a, and calculating a depth for each unit region (window region W) in the depth map using a captured image f acquired by an image sensor 13. In the step of forming the coded aperture 14a, an aperture pattern B·B4 is formed, the aperture pattern B·B4 including a first region (transmissive region) R1 having a first light transmittance T1, a second region (semi-transmissive region) R2 having a second light transmittance T2 lower than the first light transmittance T1, and a third region (shielding region) R3 having a third light transmittance T3 lower than the second light transmittance T2.

[0086] (10) The program proposed in the present disclosure causes a computer to function as a means for forming the coded aperture 14a using the liquid crystal shutter 14, a means for capturing light passing through the coded aperture 14a, and a means for calculating a depth for each unit region in a depth map using a captured image f acquired by the image sensor 13. The means for forming the coded aperture 14a forms an aperture pattern B·B4 including a first region (transmissive region) R1 having a first light transmittance T1, a second region (semi-transmissive region) R2 having a second light transmittance T2 lower than the first light transmittance T1, and a third region (shielding region) R3 having a third light transmittance T3 lower than the second light transmittance T2. [Explanation of symbols]

[0087] 10: generation system, 11: control unit, 11a: image acquisition unit, 11b: aperture control unit, 11c: depth calculation unit, 12: storage unit, 13: image sensor, 14: liquid crystal shutter, 14a: coding aperture, 15: lens, 16: input unit, B·B1 to B4: aperture patterns, D: reference depth, D 0_i : Estimated depth, E n E n_i : Energy function, F F 1 F 2 F n : Frequency characteristic of the reference depth restored image, K 1 ·K 2 ···K n : frequency characteristic of point spread function, N: imaging system, R1: transparent region, R1': semi-transparent region, R2: semi-transparent region, R2': semi-transparent region, R21a·R21b: transparent region, R22a·R22b: semi-transparent region, R23a·R23b: shielded region, R3·R3': shielded region, T1~T3: light transmittance, W·W i : window area, f n : Reference depth restored image.

Claims

1. a liquid crystal shutter forming a coded aperture; an imager that captures light transmitted through the coded aperture; a depth calculation unit that calculates a depth for each unit area in a depth map by using a captured image acquired by the image sensor; having The coded aperture has an aperture pattern including a first region having a first light transmittance, a second region having a second light transmittance lower than the first light transmittance, and a third region having a third light transmittance lower than the second light transmittance. Depth map generation system.

2. the first light transmittance is an upper limit of the light transmittance that can be realized by the liquid crystal shutter, the third light transmittance is a lower limit of the light transmittance that can be realized by the liquid crystal shutter, The second light transmittance is between the first light transmittance and the third light transmittance. The depth map generation system of claim 1 .

3. When the first transmittance is 100% and the third transmittance is 0%, the second transmittance is 40% to 60%. The depth map generation system of claim 1 .

4. When the first transmittance is 100% and the third transmittance is 0%, the second transmittance is substantially 50%. The depth map generation system of claim 1 .

5. The first region and the second region are circular and have the same center. The depth map generation system of claim 1 .

6. The aperture pattern of the coded aperture comprises a plurality of the first regions. The depth map generation system of claim 1 .

7. The aperture pattern of the coded aperture comprises a plurality of the second regions. A depth map generation system according to claim 1 or 6.

8. The depth calculation unit is Obtaining a plurality of PSFs corresponding to a plurality of discretely set reference depths and corresponding to the aperture pattern; Calculating a deviation value representing a degree of deviation between the reference depth and an actual depth for each unit area of ​​the depth map for a plurality of reference depth restored images reconstructed from the captured image using the plurality of PSFs, respectively; Calculating a depth for each unit area of ​​the depth map based on the deviation value. The depth map generation system of claim 1 .

9. forming a coding aperture utilizing a liquid crystal shutter; capturing light through the coded aperture; and Calculating a depth for each unit area in a depth map using a captured image acquired by the imaging element; Including, In the step of forming the coded opening, an opening pattern is formed including a first region having a first light transmittance, a second region having a second light transmittance lower than the first light transmittance, and a third region having a third light transmittance lower than the second light transmittance. How the depth map is generated.

10. A means for forming a coded aperture utilizing a liquid crystal shutter; means for capturing light through said coded aperture; and A means for calculating a depth for each unit area in a depth map by using a captured image acquired by the imaging element; The computer functions as The means for forming the coded aperture forms an aperture pattern including a first region having a first light transmittance, a second region having a second light transmittance lower than the first light transmittance, and a third region having a third light transmittance lower than the second light transmittance. program.

Images