Apparatus and method for generating depth map and program
The depth map generation device addresses the issue of inconsistent noise removal across varying image contrasts by dividing the image into partial regions and applying tailored noise removal parameters, enhancing the accuracy of the depth map generation process.
Patent Information
- Application Number
- JP2023083130
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-19
- Publication Date
- 2024-11-29
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing depth map generation methods fail to accurately remove noise from images with varying contrast levels, leading to incorrect noise removal in areas with different properties.
A depth map generation device that divides an image into multiple partial regions and applies different noise removal parameters to each region based on the accuracy evaluation of each partial area, using coded apertures to calculate depth and remove noise effectively.
This approach ensures that noise is accurately removed from the depth map, preventing the exclusion of valid depth estimates and improving the accuracy of the generated depth map.
Smart Images

Figure 2024166789000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a depth map generating device, a generating method, and a program. [Background technology]
[0002] Non-Patent Document 1 proposes a technology for generating a depth map from an image captured through a coded aperture. In this document, two coded apertures with different aperture patterns (shapes of light-transmitting and light-shielding regions) are used. By using two coded apertures in combination, it is possible to prevent the occurrence of frequency bands with a power spectrum of 0 in a filter (a filter for generating a restored image) used in the process of generating a depth map. [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] C.Zhou, S.Lin, and S.Nayar: Coded Aperture Pairs for Depth from Defocus, IEEE international conference on computer vision, 2009 Summary of the Invention [Problem to be solved by the invention]
[0004] In the process of generating the map, a noise judgment is required to judge whether or not a dot (pixel) in the depth map is a dot that represents the distance to the subject. Then, dots that are judged to be noise need to be removed from the depth map. However, if the same noise removal process is applied to the entire area of an image that contains multiple areas with different characteristics, such as areas with low contrast and areas with high contrast, noise removal may not be performed correctly. [Means for solving the problem]
[0005] The present disclosure provides a depth map generating device that generates a depth map from a capture image captured through a coded aperture. The generating device includes a depth calculation unit that calculates a depth for each of a plurality of unit regions that constitute the depth map, a division unit that divides an image region corresponding to the capture image into a plurality of partial regions including at least a first partial region and a second partial region, and a noise removal unit that removes noise from the depths calculated for the plurality of unit regions included in the first partial region by using a first parameter and removes noise from the depths calculated for the plurality of unit regions included in the second partial region by using a second parameter.
[0006] A depth map generation method proposed in the present disclosure is a method for generating a depth map from a capture image captured through a coded aperture. The generation method includes a depth calculation step of calculating a depth for each of a plurality of unit regions constituting the depth map, a division step of dividing an image region corresponding to the capture image into a plurality of partial regions including at least a first partial region and a second partial region, and a noise removal step of removing noise from the depths calculated for the plurality of unit regions included in the first partial region using a first parameter and removing noise from the depths calculated for the plurality of unit regions included in the second partial region using a second parameter.
[0007] A program proposed in the present disclosure is a program that causes a computer to function as an apparatus that generates a depth map from a capture image captured through a coded aperture. The program causes a computer to function as a depth calculation means that calculates a depth for each of a plurality of unit areas that constitute the depth map, a division means that divides an image area corresponding to the capture image into a plurality of partial areas including at least a first partial area and a second partial area, and a noise removal means that removes noise from the depths calculated for the plurality of unit areas included in the first partial area by using a first parameter and removes noise from the depths calculated for the plurality of unit areas included in the second partial area by using a second parameter.
[0008] The generating device, generating method, and program proposed in the present disclosure can suppress the problem of estimated depths that should actually be displayed being removed from a depth map. [Brief description of the drawings]
[0009] [Figure 1] 1 is a cross-sectional view showing an example of an imaging system of a depth map generating device. [Diagram 2] FIG. 2 is a block diagram showing hardware of a depth map generating device. [Diagram 3] 2 is a block diagram showing functions of a control unit included in the depth map generating device. FIG. [Figure 4A] FIG. 11 is a diagram showing an example of a first aperture pattern of a coded aperture portion. [Figure 4B] FIG. 13 is a diagram showing an example of a second aperture pattern of a coded aperture portion. [Diagram 5] FIG. 4 is a flow chart illustrating an example of a process executed by a control unit. [Figure 6] FIG. 13 is a diagram for explaining a deviation value map. [Figure 7] 11A and 11B are diagrams for explaining a calculation process of an estimated depth. [Figure 8] FIG. 11 is a diagram for explaining a noise removal process. [Figure 9A]10A to 10C are diagrams for explaining an example of a process for dividing an image area corresponding to a captured image. [Figure 9B] FIG. 13 is a diagram illustrating an example of a depth map. [Figure 10] 13A and 13B are diagrams for explaining a modified example of the noise removal process. [Figure 11] FIG. 11 is a flowchart illustrating another example of the process executed by the control unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, a depth map generating device, a generating method, and a program proposed in this disclosure will be described.
[0011] 1 is a cross-sectional view showing an imaging system N of a generation device 10, which is an example of a depth map generation device proposed in this disclosure. As shown in the figure, the generation device 10 has a liquid crystal panel 14 and an imaging element 13.
[0012] The imaging element 13 is an image sensor such as a Complementary Metal Oxide Semiconductor (CMOS) sensor or a Charge Coupled Device (CCD).
[0013] The liquid crystal panel 14 includes a plurality of pixels. The liquid crystal panel 14 has a coded aperture 14a in a part thereof. The control unit 11, which will be described later, drives the liquid crystal of the coded aperture 14a so as to form a predetermined aperture pattern. The aperture pattern will be described in detail later.
[0014] 1, the generation device 10 has a lens 15 disposed between the coded aperture 14a and the imaging element 13. Light that has passed through the coded aperture 14a and the lens 15 enters the imaging element 13.
[0015] Fig. 2 is a block diagram showing hardware of the depth map generating device 10. The generating device 10 has a control unit 11, a storage unit 12, and an input unit 16 in addition to the liquid crystal panel 14, the image sensor 13, and the lens 15 shown in Fig. 1.
[0016] The control unit 11 includes at least one processor, such as a CPU (Central Processing Unit) or a GPU (Graphical Processing Unit). Image data acquired by the imaging element 13 is provided to the control unit 11. The control unit 11 uses this image data to generate a depth map indicating the distance to the subject.
[0017] The storage unit 12 includes a main storage unit and an auxiliary storage unit. For example, the main storage unit is a volatile memory such as a random access memory (RAM), and the auxiliary storage unit is a non-volatile memory such as a read only memory (ROM), an electrically erasable and programmable read only memory (EEPROM), a flash memory, or a hard disk. The control unit 11 executes a program stored in the storage unit 12 to control the liquid crystal panel 14 and calculate the depth (distance to the subject). The processing executed by the control unit 11 will be described later. The generating device 10 may be, for example, a mobile device such as a smartphone or a tablet personal computer (PC), or a personal computer connected to a camera.
[0018] The input unit 16 may be a touch sensor attached to an image display device that displays a captured image. The input unit 16 may also be a pointing device such as a keyboard or a mouse. The input unit 16 inputs a signal to the control unit 11 according to a user's operation.
[0019] [Function block] 3 is a block diagram showing functions of the control unit 11. The control unit 11 has, as its functions, an image acquisition unit 11a and a depth calculation unit 11c. The image acquisition unit 11a includes an aperture control unit 11b. The depth calculation unit 11c includes a division unit 11d and a noise removal unit 11e. These functions are realized by the control unit 11 operating in accordance with a program stored in the storage unit 12.
[0020] [Aperture control section] The aperture control unit 11b controls the liquid crystal of the coded aperture 14a of the liquid crystal panel 14 so as to form a predetermined aperture pattern. Figures 4A and 4B are diagrams showing examples of aperture patterns formed by the coded aperture 14a. In these figures, the white regions are light-transmitting regions R1, and the black regions are light-shielding regions R2 and R3.
[0021] In the aperture patterns B1 and B2 shown in FIG. 4A and FIG. 4B, a circular light-transmitting region R1 is formed inside a rectangular light-shielding region R3. A circular light-shielding region R2 is formed inside the light-transmitting region R1. The light-shielding region R2 of the first aperture pattern B1 and the light-shielding region R2 of the second aperture pattern B2 are formed at different positions. In the example shown in the figure, these two light-shielding regions R2 are formed at positions that are line-symmetric. According to these two aperture patterns B1 and B2, as described in Non-Patent Document 1, it is possible to suppress the occurrence of a frequency band in which the power spectrum becomes 0 in the spatial frequency characteristics of the point spread function corresponding to the aperture pattern. Such aperture patterns B1 and B2 can be searched for, for example, by a genetic algorithm. The aperture pattern formed by the aperture control unit 11b is not limited to the example shown in FIG. 4A and FIG. 4B.
[0022] [Image acquisition section] The image acquisition unit 11a controls the image sensor 13 and the coded aperture 14a to continuously capture two images f1 and f2 using two aperture patterns B1 and B2. (Hereinafter, the images f1 and f2 are referred to as captured images.) At this time, the interval between the imaging using the first aperture pattern B1 and the imaging using the second aperture pattern B2 may be several hundreds of milliseconds or several tens of milliseconds.
[0023] The number of aperture patterns formed by the aperture control unit 11b may be more than 2. In this case, the image acquisition unit 11a may continuously capture images for the number of aperture patterns that is more than 2. The aperture control unit 11b switches between the multiple aperture patterns in sequence in synchronization with the reception of light by the imaging element 13.
[0024] [Depth calculation part] The depth calculation unit 11c generates a depth map from the captured images f1 and f2 obtained by the image acquisition unit 11a. The depth calculation unit 11c calculates the depth of the captured images f1 and f2 (the distance from the imaging system N to the subject) for each of the multiple unit areas constituting the depth map. The unit area of the depth map may be, for example, a pixel of the captured images f1 and f2. Alternatively, the unit area of the depth map may be larger than the pixel of the captured image. For example, a multiple number of adjacent pixels (e.g., 2×2) may be the unit area.
[0025] Fig. 5 is a flow diagram showing an example of the process of the depth calculation unit 11c. The process of the depth calculation unit 11c shown in Fig. 5 is started after the capture images f1 and f2 are acquired by the image acquisition unit 11a.
[0026] First, the depth calculation unit 11c performs a two-dimensional Fourier transform on the captured images f1 and f2 (S101). Hereinafter, the frequency characteristics of the captured images f1 and f2 obtained by the two-dimensional Fourier transform are represented as F1 and F2. If high-frequency components are included in the images, the effect of noise may become excessive when calculating the reference depth restored image. Therefore, the high-frequency components may be removed from the frequency characteristics F1 and F2. For example, a low-pass filter that transmits only frequencies equal to or lower than half the sampling frequency (Nyquist frequency) may be used. Then, the frequency characteristics F1 and F2 from which the high-frequency components have been removed may be used in the processing described below.
[0027] [Generating reference depth restored images] In the generating device 10, a plurality of point spread functions (PSFs) corresponding to a plurality of reference depths set discretely are prepared. The reference depths are candidate values for the distance to the subject, such as 100 mm, 300 mm, and 700 mm. Each PSF has a shape corresponding to the aperture pattern B1 / B2 of the coded aperture 14a. In addition, each PSF has a size according to the corresponding reference depth. Specifically, the size of the PSF decreases as the reference depth increases. The plurality of PSFs are stored in the storage unit 12 as frequency characteristics obtained by performing a two-dimensional Fourier transform on them. The depth calculation unit 11c acquires the PSFs as frequency characteristics from the storage unit 12. This frequency characteristic is also called an optical transfer function (OTF).
[0028] The depth calculation unit 11c uses the captured images f1 and f2 and the PSF to generate a restored image corresponding to each of a plurality of reference depths (S102). Hereinafter, this restored image is referred to as a "reference depth restored image." Specifically, the depth calculation unit 11c uses the following Equation 1 to calculate the frequency characteristics of the reference depth restored image corresponding to the frequency characteristics F1 and F2 of the captured images f1 and f2 and the frequency characteristics of the PSF.
number
[0029] Equation 1 is a generalization of the Wiener filter so that it can be applied to two aperture patterns B1 and B2. In equation 1, each letter represents the following element: F 0_d : Reference depth-reconstructed image in frequency domain F1: Frequency characteristic of captured image f1 obtained by first aperture pattern B1 F2: Frequency characteristic of captured image f2 obtained by second aperture pattern B2 ·K 1_d : Frequency characteristic (optical transfer function) of PSF corresponding to the size of the reference depth (d) having a shape corresponding to the first aperture pattern B1 ·K 2_d: Frequency characteristic (optical transfer function) of PSF corresponding to the size of the reference depth (d) having a shape corresponding to the second aperture pattern B2 ·C: SNR matrix, a regularization term that takes into account noise due to fluctuations. For example, it can be calculated using the variance of image noise σ and the frequency distribution of natural images S. K with a line 1_d· K 2_d : Frequency characteristic of PSF K 1_d· K 2_d Complex conjugate of
[0030] The size of the PSF becomes smaller as the distance (depth) from the imaging system N to the subject increases. Therefore, the frequency characteristic K 1_d ·K 2_d is also determined according to the distance from the imaging system N to the subject. In equation 1, the subscript "d" added to K1·K2 corresponds to a reference depth such as 100 mm or 300 mm. For example, the function K 1_100 ·K 2_100 is the frequency characteristic of the point spread function for an object at a distance of 100 mm from the imaging system N. The number of reference depths may be more than two, for example, 10, 20, 30, etc.
[0031] In S102, the depth calculation unit 11c calculates the frequency characteristics F1 and F2 of the captured images f1 and f2 and the frequency characteristic K 1_d ·K 2_d The frequency characteristic F of the reference depth restored image according to 0_d The depth calculation unit 11c calculates the frequency characteristic F 0_d For example, the depth calculation unit 11c calculates the frequency characteristic K 1_100 ·K 2_100 Based on the frequency characteristics F1·F2 of the captured images f1·f2, the frequency characteristic F of the reference depth restored image is calculated. 0_100 Calculate.
[0032] According to Equation 1, an object whose distance from the imaging system N is equal to the reference depth appears without blur in the reference depth restored image. For example, an object located 100 mm from the imaging system N has a frequency characteristic K 1_100 ·K 2_100 The reference depth restored image obtained by the above method appears without blur. On the other hand, the frequency characteristics K 1_d ·K 2_d In the reference depth restored image obtained by the above, blur (pixel value deviation) appears for this object. The blur becomes stronger as the difference between the actual depth of the object and the reference depth becomes larger. Therefore, in the subsequent processing, the depth calculation unit 11c uses the degree of this blur to calculate the distance (depth) to the object.
[0033] [Calculation of deviation value map] The depth calculation unit 11c calculates the frequency characteristic F 0_d Based on the frequency characteristics F1 and F2 of the captured images f1 and f2, a deviation value map M d (S103). The deviation value map M d is calculated for each of the multiple reference depths (d). d In this figure, the degree of deviation (deviation value) between the reference depth (d) and the actual depth is shown for each pixel (unit area of the depth map).
[0034] The depth calculation unit 11c calculates the deviation value map M d Calculate.
number
[0035] In number 2, each letter represents the following element: M d : Deviation value map for reference depth (d) ·IFFT: 2-dimensional inverse Fourier transform F 0_d: Frequency characteristics of the reference depth restored image for the reference depth (d) ·K 1_d : Frequency characteristic (optical transfer function) of PSF corresponding to the size of the reference depth (d) having a shape corresponding to the first aperture pattern B1 ·K 2_d : Frequency characteristic (optical transfer function) of PSF corresponding to the size of the reference depth (d) having a shape corresponding to the second aperture pattern B2 F1: Frequency characteristic of captured image f1 obtained by first aperture pattern B1 F2: Frequency characteristic of captured image f2 obtained by second aperture pattern B2
[0036] Figure 6 shows the deviation value map M d As shown in the figure, the deviation value map M d indicates the deviation value w between the actual depth (the actual distance from the imaging system N to the subject) and the reference depth for each pixel. For example, the deviation value map M 100 indicates the deviation value w between the actual depth and the reference depth (100 mm). If the actual depth is 100 mm, the deviation value map M 100 The deviation value w 100 is effectively 0, and if the actual depth is 200 mm, the deviation value map M corresponding to the reference depth (100 mm) 100 In this case, the deviation value w 100 will be greater than 0.
[0037] In this way, in the process of S102, the depth calculation unit 11c calculates the frequency characteristic K 1_d ·K 2_d Using the multiple PSFs expressed by 0_d ) is generated. Then, in S103, the frequency characteristic F 0_d Using multiple reference depth restored images represented by and captured images f1 and f2 represented by frequency characteristics F1 and F2, the deviation value w between the reference depth and the actual depth is calculated. d Calculate.
[0038] [Calculating the depth of each unit area] The depth calculation unit 11c calculates a plurality of deviation value maps M d , and calculates the depth (distance from the imaging system N to the subject) for each pixel (unit area of the depth map) (S104). FIG. 7 is a diagram for explaining this process by the depth calculation unit 11c. In FIG. 7, the horizontal axis is the reference depth, and the vertical axis is the deviation value w d Hereinafter, the depth calculated by the depth calculation unit 11c will be referred to as an estimated depth d0.
[0039] Deviation value w d is smallest at a reference depth (d) that is the same as or close to the actual depth. In the example of FIG. 7, the deviation value w d Therefore, the subject appearing in this pixel is estimated to be located at a position of 2000 mm from the imaging system N. Therefore, in S104, the depth calculation unit 11c calculates the deviation value w d The reference depth for which is minimum is searched for, and the obtained reference depth is set as the estimated depth d0. The depth calculation unit 11c executes this search for all pixels (all unit areas of the depth map), thereby obtaining a depth map before noise removal, which will be described later.
[0040] The calculation process of the depth map is not limited to the example described here. In the above process, the depth calculation unit 11c calculates the deviation value w d The deviation value map M that minimizes d The reference depth (d) of the pixel is set as the estimated depth d0 of the pixel. d A function showing the relationship between the pixel depth and the pixel depth may be calculated, and the minimum value of the function may be calculated. The depth at which the minimum value is obtained may be calculated as the estimated depth d0 of the pixel.
[0041] This process by the depth calculation unit 11c can be performed, for example, as follows. The depth calculation unit 11c calculates the reference depth (d) and the deviation value wd That is, the depth calculation unit 11c fits, for example, a cubic function to the relationship between the deviation value w d The following equation 3 is fitted to the point where d is the minimum and to multiple nearby points to find the coefficients a1·a2, a3, and a4. In equation 3, the function W(d) is a cubic function with coefficients a1·a2, a3, and a4.
number
[0042] Then, the depth calculation unit 11c obtains the minimum of Equation 3. That is, the depth calculation unit 11c calculates the depth that minimizes the function W(d) by solving ΔW / Δd=0. Then, the depth calculation unit 11c sets the depth calculated in this way as the estimated depth d0. According to such processing, it is possible to increase the resolution regarding the depth without increasing the number of reference depths (the number of PSFs).
[0043] [Accuracy assessment value] According to the above-mentioned formula 2, the estimated depth d0 can be calculated at the position where blurring occurs by performing convolution integration with the PSF. For example, the estimated depth d0 can be calculated at the boundary between the outer edge of the subject and the background (the position where a change in pixel value appears). On the other hand, it is difficult to calculate an accurate depth in an area where blurring does not occur even if convolution integration is performed with the PSF. In an area where blurring does not occur, the change in pixel value is small (contrast is low), so the deviation value w d In other words, the range of change in the deviation value w d The estimated depth d0 obtained for a pixel with a small change in the depth may be noise. Therefore, the depth calculation unit 11c determines that such an estimated depth d0 is noise and removes it from the depth map obtained in S104. The process of the depth calculation unit 11c is performed, for example, as follows.
[0044] The depth calculation unit 11c calculates an accuracy evaluation value indicating the accuracy of the estimated depth d0 obtained for each pixel (unit area of the depth map) by multiplying the deviation value w dFor example, the depth calculation unit 11c calculates the depth based on a plurality of deviation values w d The depth calculation unit 11c calculates such an accuracy evaluation value for all pixels based on the range of change in the depth. The accuracy evaluation value is expressed by, for example, the following Equation 4.
number
[0045] The change Δw is the deviation value w d For example, as shown in FIG. 7, the deviation value w d As shown in Equation 4, the depth calculation unit 11c calculates a plurality of deviation values w d The evaluation value is calculated based on the difference between the maximum and minimum of .
[0046] In general, due to the influence of noise, etc., erroneous distance measurement results are often obtained for objects at close range. Therefore, in Equation 4, such objects may be weighted by the estimated depth d0 in order to evaluate the change amount ΔW as small (to make it easier to remove as noise).
[0047] The depth calculation unit 11c determines the estimated depth d0 obtained for a pixel whose accuracy evaluation value β is lower than the noise judgment threshold as noise and removes it from the depth map. In other words, the depth calculation unit 11c displays only the estimated depth d0 obtained for a pixel whose accuracy evaluation value β is higher than the noise judgment threshold on the depth map.
[0048] 8 is a diagram for explaining this process by the noise removal unit 11e. In the diagram, the horizontal axis indicates the X-direction coordinate of the depth map, and the vertical axis indicates the accuracy evaluation value β. In this diagram, the deviation value w dSince the change amount Δw of the pixel group is relatively large, the accuracy evaluation value β of this pixel group is higher than the noise judgment threshold Nth. On the other hand, in the pixel group of region R2, the deviation value w d Since the amount of change Δw in is relatively small, the accuracy evaluation value β is lower than the noise determination threshold Nth. In this case, the depth calculation unit 11c displays the depth d0 calculated for the pixel group in region R1 on the depth map, but does not display the depth d0 calculated for the pixel group in region R2 on the depth map.
[0049] When performing the process shown in FIG. 8, how to set the noise determination threshold value Nth becomes an issue. A value obtained by multiplying the maximum value of the accuracy evaluation values β calculated for all pixels by a predetermined ratio (for example, 50%) can be considered as an example of the noise determination threshold value Nth. However, when such a noise determination threshold value Nth is used, a problem occurs in that the estimated depth d0 that should be displayed in the depth map is also determined as noise. For example, when a captured image has partially dark areas, the deviation value w in the dark areas d Therefore, the accuracy evaluation value β calculated for that region becomes smaller than the noise determination threshold Nth, and as a result, the estimated depth d0 for that region is removed from the depth map.
[0050] Therefore, the depth calculation unit 11c divides the image area corresponding to the captured image into a plurality of partial areas. Then, the noise reduction process is performed for each of the partial areas by using a plurality of mutually different parameters. This noise reduction process will be described in more detail below.
[0051] The dividing unit 11d divides an image region corresponding to the captured images f1 and f2 into a plurality of partial regions. For example, the dividing unit 11d divides the pre-noise removal depth map calculated in S104 into a plurality of partial regions (S105). FIG. 9A is a diagram for explaining an example of processing by the dividing unit 11d. In the diagram, a captured image that is the source of the depth map is shown. As shown in FIG. 9A, the dividing unit 11d divides the pre-noise removal depth map into, for example, a plurality of predefined partial regions. More specifically, the dividing unit 11d divides the pre-noise removal depth map into a plurality of rectangular partial regions having a predefined size.
[0052] [Noise Reduction] The noise removal unit 11e removes noise from the estimated depth d0 calculated for the pixels included in the first partial region A1 (see FIG. 9A) by using a first parameter set for the first partial region A1. Similarly, the noise removal unit 11e removes noise from the estimated depth d0 calculated for the pixels included in the second partial region A2 (see FIG. 9A) by using a second parameter set for the second partial region A2. This process is performed for each of the multiple partial regions A1 to A15 (see FIG. 9A). According to this process, noise removal is performed by using parameters suitable for each partial region. As a result, it is possible to suppress the problem that the estimated depth d0 that should be displayed is removed from the depth map.
[0053] The process of the noise removal unit 11e is performed, for example, as follows. After the division unit 11d executes the division process, the noise removal unit 11e first sets a parameter for noise judgment based on a plurality of accuracy evaluation values β calculated for pixels in each partial region (S106). For example, the noise removal unit 11e calculates the maximum value of the accuracy evaluation values β calculated for a plurality of pixels in each partial region. Then, the noise removal unit 11e multiplies the maximum value of the accuracy evaluation values β by a predetermined rate (A%), and sets the multiplication result (β×A%) as a parameter (noise judgment threshold) used in noise judgment for that partial region. The noise removal unit 11e calculates a noise judgment threshold for each of a plurality of partial regions A1 to A15 (see FIG. 9A). The method of calculating the noise judgment threshold may be changed as appropriate.
[0054] Next, the noise removal unit 11e compares a parameter (noise judgment threshold) set for each partial region with the accuracy evaluation value β of each pixel included in each partial region, and removes the estimated depth d0 determined to be noise based on the comparison result from the depth map (S107). For example, the noise removal unit 11e determines the estimated depth d0 calculated for a pixel whose accuracy evaluation value β is lower than the noise judgment threshold as noise, and removes it from the depth map. The noise removal unit 11e executes the noise removal process of S107 for each of the partial regions A1 to A15 (see FIG. 9A). As a result, a depth map as exemplified in FIG. 9B is obtained.
[0055] In this way, the noise removal unit 11e determines whether the estimated depth d0 of a pixel in each partial region is noise or not based on the accuracy evaluation value β calculated for each pixel in the multiple partial regions and the parameter (noise determination threshold) set for each partial region A1 to A15.
[0056] The process of the noise removal unit 11e is not limited to the example described here. For example, the parameter set for each of the partial regions A1 to A15 may not be the noise judgment threshold. FIG. 10 is a diagram for explaining a modified example of the noise removal unit 11e. As shown in FIG. 10, the noise removal unit 11e may multiply, for example, the change amount Δw of the deviation value w calculated for each pixel of the first partial region A1 (see FIG. 9A) by the first parameter α1 set for the first partial region A1. Then, the noise removal unit 11e may further multiply the change amount Δw of the deviation value w calculated for each pixel of the first partial region A1 by the estimated depth d0 calculated for each pixel, and the multiplication result (Δw×d0×α1) may be set as an accuracy evaluation value β. Then, the accuracy evaluation value β may be compared with a noise judgment threshold Nth common to a plurality of partial regions. If the accuracy evaluation value β is lower than the noise judgment threshold, the estimated depth d0 of the pixel may be removed from the depth map. The noise removal unit 11e may perform such a process for each of the plurality of partial regions A1 to A15.
[0057] In this case, the parameters α1 to α 15 may be set based on the contrast of the partial region. For example, a relatively large parameter may be used in a partial region with low contrast (e.g., a partial region in which the entire partial region is dark). More specifically, the parameter of a partial region with low contrast (e.g., a partial region in which the entire partial region is dark) may be set so that the maximum deviation value Δw of a plurality of pixels in this partial region matches the maximum deviation value Δw of pixels in a partial region with relatively high contrast. This processing can also suppress the problem that the estimated depth d0 that should be displayed is removed from the depth map.
[0058] [summary] (1) In the generating device 10, the depth calculation unit 11c calculates the depth for each of a plurality of unit areas (e.g., pixels) constituting a depth map. The division unit 11d divides an image area corresponding to a captured image into a plurality of partial areas A1 to A15. The noise removal unit 11e removes noise from the estimated depth d0 calculated for a plurality of unit areas included in the first partial area A1 by using a first parameter. Moreover, the noise removal unit 11e removes noise from the estimated depth d0 calculated for a plurality of unit areas included in the second partial area A2 by using a second parameter. This generating device 10 can prevent the problem that the estimated depth d0 that should be displayed is removed from the depth map.
[0059] (2) In (1), the noise removal unit 11e calculates an accuracy evaluation value β indicating the accuracy of the estimated depth d0 calculated for each unit region. Then, the noise removal unit 11e determines whether the estimated depth d0 calculated for each unit region in the first partial region A1 is noise or not based on the accuracy evaluation value β calculated for each unit region in the first partial region A1 and the first parameter. Also, the noise removal unit 11e determines whether the estimated depth d0 calculated for each unit region in the second partial region A2 is noise or not based on the accuracy evaluation value β calculated for each unit region in the second partial region A2 and the second parameter. This makes it possible to remove the estimated depth d0 with low accuracy from the depth map.
[0060] (3) In (2), the first parameter and the second parameter are noise determination thresholds. The noise removal unit 11e determines whether the estimated depth d0 calculated for each unit area in the first partial area A1 is noise or not based on a comparison result between the accuracy evaluation value calculated for each unit area in the first partial area A1 and the first parameter. The noise removal unit 11e also determines whether the estimated depth d0 calculated for each unit area in the second partial area A2 is noise or not based on a comparison result between the accuracy evaluation value calculated for each unit area in the second partial area A2 and the second parameter. This makes it possible to remove the estimated depth d0 with a low accuracy evaluation value from the depth map.
[0061] (4) In (2) or (3), the noise removal unit 11e sets a first parameter based on a plurality of accuracy evaluation values β calculated for a plurality of unit areas in the first partial area A1. Similarly, the noise removal unit 11e sets a second parameter based on a plurality of accuracy evaluation values β calculated for a plurality of unit areas in the second partial area A2. This allows the parameters to be different between a partial area where pixels with low accuracy evaluation values are concentrated and a partial area where pixels with high accuracy evaluation values are concentrated.
[0062] (5) In (1) to (4), the depth calculation unit 11c obtains a plurality of PSFs corresponding to a plurality of discretely set reference depths and corresponding to the aperture pattern of the coding aperture. Then, the depth calculation unit 11c calculates a deviation value w , which represents the degree of deviation between the reference depth and the actual depth, based on a plurality of reference depth restored images reconstructed from the captured image using the plurality of PSFs, respectively, and the captured image. d The depth calculation unit 11c calculates the deviation value w d Based on this, an estimated depth d0 is calculated for each unit region (S104).
[0063] (6) In (5), the noise removal unit 11e calculates an accuracy evaluation value β indicating the accuracy of the estimated depth d0 calculated for each unit region by multiplying the deviation value w d Calculation is based on.
[0064] (7) In (5) or (6), the noise removal unit 11e subtracts the plurality of deviation values w calculated for each unit region from the plurality of deviation values w d The accuracy evaluation value β is calculated based on the range of change due to the reference depth.
[0065] (8) In (5) to (7), the noise removal unit 11e sets a first parameter based on a plurality of accuracy evaluation values β calculated for a plurality of unit areas in the first partial area A1, and sets a second parameter based on a plurality of accuracy evaluation values β calculated for a plurality of unit areas in the second partial area A2.
[0066] [Other examples] Note that the depth map generating device, generating method, and program proposed in this disclosure are not limited to the above-mentioned examples.
[0067] In the above-described example, as shown in Fig. 4, after calculating the estimated depth d0 for each pixel, the image area corresponding to the captured images f1 and f2 is divided into a plurality of partial areas A1 to A15. However, differently from this, the control unit 11 may divide the image area corresponding to the captured images f1 and f2 into a plurality of partial areas A1 to A15 before calculating the estimated depth d0. Then, the processing of S101 to S104 shown in Fig. 4 may be performed for each of the divided partial areas A1 to A15.
[0068] FIG. 11 is a flow diagram showing an example of such processing.
[0069] First, after capture images f1 and f2 are acquired, the division unit 11d divides each of the capture images f1 and f2 into a plurality of partial regions A1 to A15 (S201). The partial regions A1 to A15 may be rectangular regions of a predetermined size, for example, as described with reference to FIG. 9A. Next, the depth calculation unit 11c performs a two-dimensional Fourier transform on each of the plurality of partial regions A1 to A15 to calculate the frequency characteristics of each partial region A1 to A15 (S202). Next, the depth calculation unit 11c uses Equation 1 to calculate the image of each partial region expressed by the frequency characteristics and the frequency characteristics K 1_d ·K 2_d The depth calculation unit 11c generates a reference depth restored image corresponding to the PSF represented by the PSF for each of the multiple reference depths (d) (S203). Then, the depth calculation unit 11c uses Equation 2 to calculate the frequency characteristic F 0_d Based on the frequency characteristics F1 and F2 obtained for the partial regions of the captured images f1 and f2, a deviation value map M dThe depth calculation unit 11c calculates the deviation value maps M obtained for each of the partial regions A1 to A15. d For example, the depth calculation unit 11c calculates an estimated depth d0 for each pixel by referring to the deviation value w d The deviation value map M d The reference depth of is set to the estimated depth d0 of that pixel. This generates a depth map before noise removal.
[0070] The noise removal unit 11e sets a parameter (noise determination threshold) for noise determination based on the accuracy evaluation value β calculated for the pixels in each of the partial regions A1 to A15 (S206). For example, the noise removal unit 11e multiplies the maximum value of the accuracy evaluation value β calculated for the pixels in each of the partial regions A1 to A15 by a predetermined ratio (for example, 50%) and sets the multiplication result as a parameter (noise determination threshold) for noise determination. Then, the noise removal unit 11e compares the parameter (noise determination threshold) set for each of the partial regions A1 to A15 with the accuracy evaluation value β of the pixels included in each of the partial regions A1 to A15. Then, the estimated depth d0 determined to be noise based on the comparison result is removed from the depth map (S207). For example, the estimated depth d0 calculated for a pixel whose accuracy evaluation value β is lower than, for example, the noise determination threshold is determined to be noise and is removed from the depth map.
[0071] In the depth map generating device 10, the control unit 11 acquires two captured images f1 and f2 using two mutually different coded apertures. Alternatively, the control unit 11 may acquire three captured images using three mutually different coded apertures. As yet another example, the control unit 11 may acquire captured images using one type of coded aperture.
[0072] The control unit 11 calculates the accuracy evaluation value β of the estimated depth d0 by multiplying the deviation value w dThe calculation of the accuracy evaluation value β is not limited to this. For example, the control unit 11 may calculate the accuracy evaluation value β of the estimated depth d0 based on the difference between the maximum and minimum of the deviation value w d It may be calculated based on the minimum of
[0073] The depth calculation unit 11c may also perform various image processing on the captured images f1 and f2. For example, after the division process by the division unit 11d, the depth calculation unit 11c may calculate the contrast of each partial region. Then, if there is a partial region whose contrast is lower than a predetermined value and whose maximum brightness is lower than a predetermined value, image processing may be performed on that partial region or on the entire captured images f1 and f2 to expand the histogram. This makes it possible to calculate an estimated depth d0 even for dark partial regions. [Explanation of symbols]
[0074] 10: depth map generating device, 11: control unit, 11a: image acquisition unit, 11b: aperture control unit, 11c: depth calculation unit, 11d: division unit, 11e: noise removal unit, 12: storage unit, 13: imaging element, 14: liquid crystal panel, 14a: coded aperture, 15: lens, 16: input unit, A1 to A15: partial region, B1: first aperture pattern, B2: second aperture pattern, F 0_d : Frequency characteristics of the reference depth restored image, F1·F2: Frequency characteristics of the captured image, K 1_d ·K 2_d : Frequency characteristics of PSF, M d : deviation value map, R1: light transmission area, R2 and R3: light blocking area, d0: estimated depth, f1 and f2: captured image, w d : Deviation value, Δw: Amount of change.
Claims
1. A depth map generating device for generating a depth map from an image captured through a coded aperture, comprising: a depth calculation unit that calculates a depth for each of a plurality of unit areas that constitute the depth map; a division unit that divides an image area corresponding to the captured image into a plurality of partial areas including at least a first partial area and a second partial area; a noise removal unit that removes noise from depths calculated for a plurality of unit regions included in the first partial region by using a first parameter, and removes noise from depths calculated for a plurality of unit regions included in the second partial region by using a second parameter; 2. A depth map generator comprising:
2. The noise removal unit includes: Calculating an accuracy evaluation value indicating the accuracy of the calculated depth for each unit region; determining whether or not the depth calculated for each unit region in the first partial region is noise based on the accuracy evaluation value calculated for each unit region in the first partial region and the first parameter; Based on the accuracy evaluation value calculated for each unit region in the second partial region and the second parameter, it is determined whether or not the depth calculated for each unit region in the second partial region is noise. The apparatus for generating a depth map according to claim 1 .
3. the first parameter and the second parameter are thresholds for noise determination, The noise removal unit includes: determining whether the depth calculated for each unit region in the first partial region is noise or not based on a comparison result between the accuracy evaluation value calculated for each unit region in the first partial region and the first parameter; Based on a comparison result between the accuracy evaluation value calculated for each unit region in the second partial region and the second parameter, it is determined whether or not the depth calculated for each unit region in the second partial region is noise. The apparatus for generating a depth map according to claim 2 .
4. The noise removal unit includes: setting the first parameter based on a plurality of the accuracy evaluation values calculated for a plurality of unit regions in the first partial region; The second parameter is set based on a plurality of the accuracy evaluation values calculated for a plurality of unit regions in the second partial region. The apparatus for generating a depth map according to claim 2 .
5. The depth calculation unit is Obtaining a plurality of PSFs corresponding to a plurality of reference depths that are discretely set and corresponding to an aperture pattern of the coded aperture; calculating, for each unit region, a deviation value representing a degree of deviation between the reference depth and an actual depth based on the captured image and a plurality of reference depth restored images reconstructed from the captured image using the plurality of PSFs, respectively; Calculating a depth for each unit region based on the deviation value. The apparatus for generating a depth map according to claim 1 .
6. The noise removal unit calculates an accuracy evaluation value indicating accuracy of the depth calculated for each unit region based on the deviation value calculated for each unit region. An apparatus for generating a depth map according to claim 5 .
7. The noise removal unit calculates the accuracy evaluation value based on a range of change of the plurality of deviation values calculated for each unit region due to the reference depth. An apparatus for generating a depth map according to claim 6.
8. The noise removal unit includes: setting the first parameter based on a plurality of accuracy evaluation values calculated for a plurality of unit regions in the first partial region; The second parameter is set based on a plurality of accuracy evaluation values calculated for a plurality of unit regions in the second partial region. An apparatus for generating a depth map according to claim 7.
9. 1. A method for generating a depth map from an image captured through a coded aperture, comprising: a depth calculation step of calculating a depth for each of a plurality of unit areas constituting the depth map; a dividing step of dividing an image area corresponding to the captured image into a plurality of partial areas including at least a first partial area and a second partial area; a noise removal step of removing noise from depths calculated for a plurality of unit regions included in the first partial region by using a first parameter, and removing noise from depths calculated for a plurality of unit regions included in the second partial region by using a second parameter; How to generate a depth map, including:
10. A program for causing a computer to function as an apparatus for generating a depth map from an image captured through a coded aperture, the program comprising: a depth calculation means for calculating a depth for each of a plurality of unit areas constituting the depth map; A dividing means for dividing an image area corresponding to the captured image into a plurality of partial areas including at least a first partial area and a second partial area; a noise removing means for removing noise from depths calculated for a plurality of unit regions included in the first partial region by using a first parameter, and removing noise from depths calculated for a plurality of unit regions included in the second partial region by using a second parameter; A program that makes a computer function as a
