Apparatus and method for generating depth map and program
The depth map generation device improves depth accuracy by calculating and selecting pixels based on deviation values, addressing the low accuracy issues in existing methods.
Patent Information
- Application Number
- JP2023083131
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-19
- Publication Date
- 2024-11-29
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The depth calculation methods using coded apertures in existing technologies often result in low accuracy, with significant differences between calculated depths and actual depths.
A depth map generation device and method that utilizes a depth calculation unit to calculate depth for multiple pixels, a pixel selection unit to select depth display pixels based on deviation values between reference and actual depths, and a storage unit to store PSFs corresponding to coded aperture patterns, improving depth accuracy by selecting pixels with minimal deviation.
Enhances the accuracy of depth maps by selecting pixels with minimal deviation between reference and actual depths, resulting in more precise depth calculations.
Smart Images

Figure 2024166790000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a depth map generating device, a generating method, and a program. [Background technology]
[0002] Non-Patent Document 1 proposes a technology for generating a depth map from an image captured through a coded aperture. In this document, two coded apertures with different aperture patterns (shapes of light-transmitting and light-shielding regions) are used. By using two coded apertures in combination, it is possible to prevent the occurrence of frequency bands with a power spectrum of 0 in a filter (a filter for generating a restored image) used in the process of generating a depth map.
[0003] In Non-Patent Document 1, the depth is calculated as follows: (1) A PSF is prepared that has a size corresponding to a predefined reference depth such as 100 mm or 300 mm and corresponds to the aperture pattern of the coded aperture. (2) A restored image is generated for each of a plurality of reference depths using two captured images acquired through two coded apertures and the PSF. (3) The deviation between the grayscale value of each pixel of the captured image and the grayscale value of each pixel of the restored image is calculated, and a reference depth corresponding to a PSF that can obtain a small deviation is calculated as the depth of each pixel where the subject is displayed. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] C.Zhou, S.Lin, and S.Nayar: Coded Aperture Pairs for Depth from Defocus, IEEE international conference on computer vision, 2009 Summary of the Invention [Problem to be solved by the invention]
[0005] The depths calculated by the method disclosed in Non-Patent Document 1 may include low-precision depths (depths that differ greatly from the actual depths). [Means for solving the problem]
[0006] The depth map generating device proposed in the present disclosure is a depth map generating device that generates a depth map including a plurality of pixels each representing a depth from a capture image captured through a coded aperture. The generating device includes a depth calculation unit that calculates depth for the plurality of pixels, a pixel selection unit that selects a depth display pixel that is a pixel that indicates depth in the depth map, and a storage unit that stores a PSF that corresponds to a predefined reference depth and corresponds to an aperture pattern of the coded aperture. The pixel selection unit calculates a deviation value representing a degree of deviation between the reference depth and an actual depth for each pixel based on a reference depth restored image reconstructed from the capture image using the PSF and the capture image, and selects the depth display pixel based on a change in the deviation value of pixels aligned along a predetermined direction.
[0007] The depth map generation method proposed in the present disclosure is a depth map generation method for generating a depth map including a plurality of pixels each representing a depth from a captured image captured through a coded aperture. The generation method includes a depth calculation step of calculating depth for the plurality of pixels, a pixel selection step of selecting depth display pixels which are pixels indicative of depth in the depth map, and a step of acquiring a PSF corresponding to a predefined reference depth and corresponding to an aperture pattern of the coded aperture from a storage unit. In the pixel selection step, a deviation value representing a degree of deviation between the reference depth and an actual depth is calculated for each pixel based on a reference depth restored image reconstructed from the captured image using the PSF and the captured image, and the depth display pixel is selected based on a change in the deviation value of pixels aligned along a predetermined direction.
[0008] The program proposed in the present disclosure is a program that causes a computer to function as a depth map generating device that generates a depth map including a plurality of pixels each representing a depth from a capture image captured through a coded aperture. This program causes a computer to function as a depth calculation means that calculates depth for the plurality of pixels, a pixel selection means that selects a depth display pixel that is a pixel that indicates depth in the depth map, and a means that acquires a PSF that corresponds to a predefined reference depth and corresponds to an aperture pattern of the coded aperture from a storage unit. The pixel selection means calculates a deviation value that represents a degree of deviation between the reference depth and the actual depth for each pixel based on a reference depth restored image reconstructed from the capture image using the PSF and the capture image, and selects the depth display pixel based on a change in the deviation value of pixels aligned along a predetermined direction.
[0009] According to the depth map generating device, generating method, and program proposed in the present disclosure, it is possible to improve the accuracy of the depth displayed in the depth map. [Brief description of the drawings]
[0010] [Figure 1] FIG. 2 is a schematic diagram illustrating an example of an imaging system of a depth map generating device. [Diagram 2] FIG. 2 is a block diagram showing hardware of a depth map generating device. [Diagram 3] 2 is a block diagram showing functions of a control unit included in the depth map generating device. FIG. [Figure 4] FIG. 13 shows examples of two aperture patterns of coded apertures. [Diagram 5] 4 is a flow diagram showing an example of processing executed by a depth calculation unit, a pixel selection unit, and a generation unit shown in FIG. 3. [Figure 6] 11 is a diagram for explaining a deviation value map generated by a depth calculation unit; FIG. [Figure 7] FIG. 13 is a diagram showing an example of a capture image. [Figure 8]8 is a diagram showing changes in deviation values calculated for the two pixels illustrated in FIG. 7. FIG. [Figure 9] 13 is a diagram for explaining a depth map that displays the depth of all pixels without executing processing by a pixel selection unit. FIG. [Figure 10] 11 is a diagram for explaining pixel scanning by a pixel selection unit; FIG. [Figure 11] 11 is a diagram for explaining a process of selecting depth display pixels performed by a pixel selection unit. FIG. [Figure 12] 11 is a diagram for explaining a process of calculating the depth of a depth display pixel, which is performed by a depth calculation unit. FIG. [Figure 13] FIG. 13 is a diagram showing an example of a depth map generated as a result of the processing described with reference to FIG. 12. [Figure 14] 13A to 13C are diagrams showing modified examples of two aperture patterns of the coded aperture. [Figure 15] 15 is a diagram for explaining a process executed when a capture image is acquired through the aperture pattern shown in FIG. 14. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, a depth map generating device, a generating method, and a program proposed in this disclosure will be described.
[0012] [Depth map generator hardware] 1 is a cross-sectional view showing an imaging system N of a generation device 10, which is an example of a depth map generation device proposed in this disclosure. As shown in the figure, the generation device 10 has a liquid crystal panel 14 and an imaging element 13.
[0013] The imaging element 13 is an image sensor such as a Complementary Metal Oxide Semiconductor (CMOS) sensor or a Charge Coupled Device (CCD).
[0014] The liquid crystal panel 14 includes a plurality of pixels. The liquid crystal panel 14 has a coded aperture 14a in a part thereof. The control unit 11, which will be described later, drives the liquid crystal of the coded aperture 14a so as to form a predetermined aperture pattern. The aperture pattern will be described in detail later.
[0015] 1, the generation device 10 has a lens 15 disposed between the coded aperture 14a and the imaging element 13. Light that has passed through the coded aperture 14a and the lens 15 enters the imaging element 13.
[0016] Fig. 2 is a block diagram showing hardware of the depth map generating device 10. The generating device 10 has a control unit 11, a storage unit 12, and an input unit 16 in addition to the liquid crystal panel 14, the image sensor 13, and the lens 15 shown in Fig. 1.
[0017] The control unit 11 includes at least one processor, such as a CPU (Central Processing Unit) or a GPU (Graphical Processing Unit). Image data acquired by the imaging element 13 is provided to the control unit 11. The control unit 11 uses this image data to generate a depth map indicating the distance to the subject.
[0018] The storage unit 12 includes a main storage unit and an auxiliary storage unit. For example, the main storage unit is a volatile memory such as a random access memory (RAM), and the auxiliary storage unit is a non-volatile memory such as a read only memory (ROM), an electrically erasable and programmable read only memory (EEPROM), a flash memory, or a hard disk. The control unit 11 executes a program stored in the storage unit 12 to control the liquid crystal panel 14 and calculate the depth (distance to the subject). The processing executed by the control unit 11 will be described later. The generating device 10 may be, for example, a mobile device such as a smartphone or a tablet personal computer (PC), or a personal computer connected to a camera.
[0019] The input unit 16 may be a touch sensor attached to the liquid crystal panel 14. The input unit 16 may also be a pointing device such as a keyboard or a mouse. The input unit 16 inputs a signal to the control unit 11 in response to a user's operation.
[0020] [Processing performed by the control unit] 3 is a block diagram showing functions of the control unit 11. The control unit 11 has, as its functions, an image acquisition unit 11a, a depth calculation unit 11c, a pixel selection unit 11d, and a map generation unit 11g. The image acquisition unit 11a includes an aperture control unit 11b. These functions are realized by the control unit 11 operating in accordance with a program stored in the storage unit 12.
[0021] [Aperture control section] The aperture control unit 11b controls the liquid crystal of the coded aperture 14a of the liquid crystal panel 14 so as to form a predetermined aperture pattern. Fig. 4 shows examples of aperture patterns formed by the coded aperture 14a. In these figures, the white regions are light-transmitting regions R1, and the black regions are light-shielding regions R2 and R3.
[0022] In the aperture patterns B1 and B2 shown in FIG. 4, a circular light-transmitting region R1 is formed inside a rectangular light-shielding region R3. A circular light-shielding region R2 is formed inside the light-transmitting region R1. The light-shielding region R2 of the first aperture pattern B1 and the light-shielding region R2 of the second aperture pattern B2 are formed in different positions. In the example shown in the figure, these two light-shielding regions R2 are formed in line-symmetric positions. According to these two aperture patterns B1 and B2, as described in Non-Patent Document 1, it is possible to suppress the occurrence of a frequency band in which the power spectrum becomes 0 in the spatial frequency characteristics of the point spread function (PSF) corresponding to the aperture pattern. Such aperture patterns B1 and B2 can be searched for, for example, by a genetic algorithm.
[0023] The image acquisition unit 11a controls the image sensor 13 and the coded aperture 14a to continuously capture two images f1 and f2 using two aperture patterns B1 and B2. (Hereinafter, the images f1 and f2 are referred to as captured images.) At this time, the interval between the imaging using the first aperture pattern B1 and the imaging using the second aperture pattern B2 may be several hundreds of milliseconds or several tens of milliseconds.
[0024] The number of aperture patterns formed by the aperture control unit 11b may be more than 2. In this case, the image acquisition unit 11a may continuously capture images for the number of aperture patterns that is more than 2. The aperture control unit 11b switches between the multiple aperture patterns in sequence in synchronization with the reception of light by the imaging element 13.
[0025] [Depth calculation part] The depth calculation unit 11c calculates the depth (the distance from the imaging system N to the subject) for each pixel constituting the depth map. Each pixel of the depth map may correspond to, for example, one pixel in the captured images f1 and f2. Alternatively, each pixel of the depth map may correspond to multiple adjacent pixels (for example, 2×2) in the captured images.
[0026] As described below, in the generation device 10, the depth calculation unit 11c calculates the depth only for the pixels (depth display pixels) selected by the pixel selection unit 11d. Then, the map generation unit 11g displays the calculated depth in the depth display pixels of the depth map. Alternatively, the depth calculation unit 11c may calculate the depth for all pixels. Then, the map generation unit 11g may display the calculated depth only for the pixels (depth display pixels) selected by the pixel selection unit 11d.
[0027] 5 is a flow diagram showing an example of the processing of the depth calculation unit 11c, the pixel selection unit 11d, and the map generation unit 11g. The processing shown in this diagram is started after the capture images f1 and f2 are acquired by the image acquisition unit 11a.
[0028] First, the depth calculation unit 11c performs a two-dimensional Fourier transform on the captured images f1 and f2 (S101). Hereinafter, the frequency characteristics of the captured images f1 and f2 obtained by the two-dimensional Fourier transform are represented as F1 and F2. If high-frequency components are included in the images, the effect of noise may become excessive when calculating the reference depth restored image. Therefore, the high-frequency components may be removed from the frequency characteristics F1 and F2. For example, a low-pass filter that transmits only frequencies equal to or lower than half the sampling frequency (Nyquist frequency) may be used. Then, the frequency characteristics F1 and F2 from which the high-frequency components have been removed may be used in the processing described below.
[0029] In the generating device 10, a plurality of point spread functions (PSFs) corresponding to a plurality of reference depths set discretely are prepared. The reference depths are candidate values for the distance to the subject, such as 100 mm, 300 mm, and 700 mm. Each PSF has a shape corresponding to the aperture pattern B1 / B2 of the coded aperture 14a. Each PSF has a size according to the reference depth. Specifically, the size of the PSF decreases as the reference depth increases. The plurality of PSFs are stored in the storage unit 12 as frequency characteristics obtained by performing a two-dimensional Fourier transform on them. The depth calculation unit 11c acquires the frequency characteristics of the PSFs from the storage unit 12. This frequency characteristic is also called an optical transfer function (OTF).
[0030] The depth calculation unit 11c uses the captured images f1 and f2 and the PSF to generate a restored image corresponding to each of a plurality of reference depths (S102). Hereinafter, this restored image is referred to as a "reference depth restored image." Specifically, the depth calculation unit 11c uses the following Equation 1 to calculate the frequency characteristics of the reference depth restored image corresponding to the frequency characteristics F1 and F2 of the captured images f1 and f2 and the frequency characteristics of the PSF.
[0031]
number
[0032] Equation 1 is a generalization of the Wiener filter so that it can be applied to two aperture patterns B1 and B2. In equation 1, each letter represents the following element: F 0_d : Reference depth-reconstructed image in frequency domain F1: Frequency characteristic of captured image f1 obtained by first aperture pattern B1 F2: Frequency characteristic of captured image f2 obtained by second aperture pattern B2 ·K 1_d : Frequency characteristic (optical transfer function) of PSF corresponding to the size of "reference depth: d" having a shape corresponding to the first aperture pattern B1 ·K2_d : Frequency characteristic (optical transfer function) of PSF corresponding to the size of "reference depth: d" having a shape corresponding to the second aperture pattern B2 ·C: SNR matrix, a regularization term that takes into account noise due to fluctuations. For example, it can be calculated using the variance of image noise σ and the frequency distribution of natural images S. K with a line 1_d· K 2_d : Frequency characteristic of PSF K 1_d· K 2_d Complex conjugate of
[0033] The size of the PSF becomes smaller as the distance (depth) from the imaging system N to the subject increases. Therefore, the frequency characteristic K 1_d ·K 2_d is also determined according to the distance from the imaging system N to the subject. In equation 1, the subscript "d" added to K1·K2 corresponds to a reference depth such as 100 mm or 300 mm. For example, the function K 1_100 ·K 2_100 is the frequency characteristic of the PSF for an object that is 100 mm away from the imaging system N. The number of reference depths may be more than two, for example, 10, 20, 30, etc.
[0034] In S102, the depth calculation unit 11c calculates the frequency characteristics F1 and F2 of the captured images f1 and f2 and the frequency characteristic K 1_d ·K 2_d The frequency characteristic F of the reference depth restored image according to 0_d The depth calculation unit 11c calculates the frequency characteristic F of the reference depth restored image for each of the multiple reference depths. 0_d For example, the depth calculation unit 11c calculates the frequency characteristic K 1_100 ·K 2_100 Based on the frequency characteristics F1·F2 of the captured images f1·f2, the frequency characteristic F of the reference depth restored image is calculated. 0_100 In addition, the depth calculation unit 11c calculates the frequency characteristic K 1_300 ·K 2_300Based on the frequency characteristics F1·F2 of the captured images f1·f2, the frequency characteristic F of the reference depth restored image is calculated. 0_300 Calculate.
[0035] According to Equation 1, an object whose distance from the imaging system N is equal to the reference depth appears without blur in the reference depth restored image. For example, an object located 100 mm from the imaging system N has a frequency characteristic K 1_100 ·K 2_100 The reference depth restored image obtained by the above method appears without blur. On the other hand, the frequency characteristics K 1_d ·K 2_d In the reference depth restored image obtained by the above, blur (shift in gradation value) appears for this object. The blur becomes stronger as the difference between the actual depth of the object and the reference depth becomes larger. Therefore, in the subsequent processing, the depth calculation unit 11c uses the degree of this blur to calculate the distance (depth) to the object.
[0036] [Deviation value map] The depth calculation unit 11c calculates the frequency characteristic F 0_d Based on the frequency characteristics F1 and F2 of the captured images f1 and f2, a deviation value map M d (S103). The deviation value map M d is calculated for each of multiple reference depths (100 mm, 300 mm, etc.). d In the , the degree of deviation (deviation value) between the reference depth and the actual depth is indicated for each pixel of the depth map.
[0037] The depth calculation unit 11c calculates the deviation value map M d Calculate.
[0038]
number
[0039] In number 2, each letter represents the following element: M d : Deviation value map for "reference depth: d" ·IFFT: 2-dimensional inverse Fourier transform F 0_d : Frequency characteristics of reference depth restored image for "reference depth: d" ·K 1_d : Frequency characteristic (optical transfer function) of PSF corresponding to the size of "reference depth: d" having a shape corresponding to the first aperture pattern B1 ·K 2_d : Frequency characteristic (optical transfer function) of PSF corresponding to the size of "reference depth: d" having a shape corresponding to the second aperture pattern B2 F1: Frequency characteristic of captured image f1 obtained by first aperture pattern B1 F2: Frequency characteristic of captured image f2 obtained by second aperture pattern B2
[0040] Figure 6 shows the deviation value map M d As shown in the figure, the deviation value map M d indicates the deviation value w between the actual depth (the actual distance from the imaging system N to the subject) and the reference depth for each pixel. For example, the deviation value map M 100 indicates the deviation value w between the actual depth and the "reference depth: 100 mm". If the actual depth is 100 mm, the deviation value map M 100 The deviation value w 100 is the smallest, and if the actual depth is 200 mm, the deviation value map M corresponding to "reference depth: 100 mm" 100 In this case, the deviation value w 100 is greater than the minimum.
[0041] In this way, in the process of S102, the depth calculation unit 11c calculates the frequency characteristic K 1_d ·K 2_d Using the multiple PSFs expressed by 0_d) is generated. Then, in S103, the frequency characteristic F 0_d The deviation value w between the reference depth and the actual depth is calculated using multiple reference depth restored images represented by and captured images f1 and f2 represented by frequency characteristics F1 and F2.
[0042] [Pixel selection section] In steps S104 to S106, the pixel selection unit 11d selects a plurality of deviation value maps M d and selects a depth display pixel. The depth display pixel is a pixel that indicates the depth (the distance from the imaging system N to the subject) in the depth map. Then, in S107, the depth calculation unit 11c calculates the depth for the depth display pixel.
[0043] Unlike the example described here, the depth calculation unit 11c may calculate the depth for all pixels, and may display the calculated depth only on the depth display pixel selected by the pixel selection unit 11d.
[0044] In the following, the processing of the pixel selection unit 11d and the depth calculation unit 11c will be described using as an example a case where the checker pattern illustrated in Fig. 7 is photographed in the processing of the image acquisition unit 11a. Fig. 8 is a diagram for explaining the processing of the pixel selection unit 11d and the depth calculation unit 11c, in which the horizontal axis indicates a reference depth such as 100 mm or 300 mm, and the vertical axis indicates the deviation value w at each pixel. In the following, the depth calculated by the depth calculation unit 11c is referred to as an estimated depth d0.
[0045] The deviation value w is smallest at a reference depth that is the same as the actual depth or a reference depth that is close to the actual depth. For example, line A in FIG. 8 shows the deviation value w calculated for a pixel in area Pa shown in FIG. 7. As shown by line A, the deviation value w changes depending on the reference depth. The pixels in area Pa are pixels located on the boundary between white and black in the checkered pattern. The deviation value w shown by line A is smallest at the position of "reference depth: 1500 mm". Therefore, the subject appearing in this pixel (the checkered pattern shown in FIG. 7) is estimated to be located 1500 mm from the imaging system N. That is, 1500 mm is calculated as the estimated depth d0 for this pixel.
[0046] However, in areas away from the boundary between black and white, the gradation values of multiple pixels are substantially the same, so blurring is unlikely to occur in the captured images f1 and f2. Therefore, in pixels in such areas, the change in deviation value w according to the reference depth is small. Therefore, in such areas, the reference depth at which the deviation value w is smallest is unlikely to match the actual depth (distance to the subject). In other words, in such areas, the error between the reference depth at which the deviation value w is smallest and the actual depth is large.
[0047] For example, line B in FIG. 8 shows the change in deviation value w calculated for an image in region Pb shown in FIG. 7 according to the reference depth. Region Pb is located at the center of the white region of the checkered pattern. The amount of change Δwb in the deviation value w shown by line B is smaller than the amount of change Δwa in the deviation value w shown by line A. In addition, since the entire checkered pattern shown in FIG. 7 is equidistant from the imaging system N, the deviation value w shown by line B should be smallest at the position of "reference depth: 1500 mm" like line A, but is smallest at the position of "reference depth: 2000 mm."
[0048] In this way, it is difficult to calculate a correct estimated depth d0 in areas where the change in gradation value is small, such as white and black areas. Therefore, as shown in the upper diagram of Figure 9, for pixels in white and black areas, a depth that is different from the actual depth of 1500 mm, such as 1000 mm or 2000 mm, is often calculated as the estimated depth d0.
[0049] Furthermore, as shown in the lower diagram of FIG. 9, even near the boundary between a white area and a black area, a depth different from the actual depth (1500 mm) may be calculated as the estimated depth.
[0050] [Selection of candidate pixels, common candidate pixels, and depth display pixels] Therefore, the pixel selection unit 11d selects a pixel from which a correct estimated depth d0 is likely to be calculated as a depth display pixel, and the depth calculation unit 11c calculates the depth for that depth display pixel.
[0051] Specifically, the pixel selection unit 11d selects each deviation value map M d With reference to the deviation value w of pixels aligned in a predetermined direction, the pixel selection unit 11d searches for a pixel with a high probability of calculating a correct estimated depth d0 based on the change in deviation value w of pixels aligned in a predetermined direction. (Hereinafter, this predetermined direction will be referred to as the pixel scanning direction.) More specifically, the pixel selection unit 11d searches for a pixel with a maximum deviation value w from among the pixels aligned in the pixel scanning direction. Then, the pixel obtained by the search is set as a candidate pixel (S104).
[0052] For example, the deviation value map M 100 Among the pixels arranged in the pixel scanning direction in the , one or more pixels with a maximum deviation value w are selected as candidate pixels. 300 The pixel selection unit 11d selects one or more pixels having a maximum deviation value w as candidate pixels from among the pixels arranged in the pixel scanning direction in the other deviation value map M d The same process is performed for
[0053] Then, the pixel selection unit 11d selects a plurality of deviation value maps M dIf multiple candidate pixels selected from the deviation value map M satisfy a predetermined condition, the pixel that satisfies the condition is selected as a depth display pixel. For example, if the number of candidate pixels at the same position is equal to or greater than a predetermined number, the pixel selection unit 11d selects the candidate pixel as one of the depth display pixels. In other words, if the number of candidate pixels at the same position is equal to or greater than a predetermined number, the pixel selection unit 11d selects the candidate pixel as one of the depth display pixels. d The pixel selected as the candidate pixel from among the pixels (common candidate pixel) is selected as one of the depth display pixels (S105).
[0054] The processing of the pixel selection unit 11d will be described in detail with reference to Figs. 10 and 11. The left diagram of Fig. 10 shows the deviation value map M 100 The center diagram in FIG. 10 shows an example of the deviation value w in a part of the region in the deviation value map M 300 Similarly, the right diagram of FIG. 10 illustrates the deviation value w in the same region. 700 10 illustrates deviation values w for the same region in FIG. 10. Also, m-2, m-1, m, etc. written on the left side of each figure in FIG. 10 are pixel row numbers. n-2, n-1, n, etc. written on the upper side of FIG. 10 are pixel column numbers. FIG. 11 shows deviation values w in each deviation value map. In this figure, the vertical axis is deviation value, and the horizontal axis is pixel row number (m, m+1, etc.).
[0055] The pixel selection unit 11d scans the deviation values w of the pixels along the pixel scanning direction. The pixel scanning direction is a direction according to the two types of opening patterns B1 and B2, specifically, the vertical direction. In S104, the pixel selection unit 11d compares the deviation values w of multiple pixels (e.g., three or more and seven or less pixels) lined up in the vertical direction with each other, and searches for a pixel (candidate pixel) with a maximum deviation value w. The pixel selection unit 11d performs this search in order from one end (the uppermost pixel) to the opposite end (the lowermost pixel) in the vertical direction of the pixel area. The pixel selection unit 11d records the found candidate pixel in the storage unit 12.
[0056] The pixel selection unit 11d may fit a function to the deviation value w of a predetermined number of pixels arranged vertically in the nth column, or to all pixels in the nth column. Then, the pixel selection unit 11d may use the derivatives of the function (the value of the first derivative and the value of the second derivative) to identify a candidate pixel where the deviation value w is maximum.
[0057] The deviation value map M shown in the left diagram of FIG. 10 and in FIG. 11 100 In this case, the deviation value w is maximum at pixel (n, m-1). Therefore, pixel selector 11d records pixel (n, m-1) in storage 12 as one of the candidate pixels.
[0058] The deviation value map M shown in the center of Fig. 10 and Fig. 11 300 In the example, the deviation value w is maximum at pixel (n, m). Therefore, the pixel selector 11d records this pixel (n, m) as one of the candidate pixels in the memory 12. Furthermore, in the deviation value map M shown in the right diagram of FIG. 700 In this case, the deviation value w is maximum at pixel (n, m), and the pixel selection unit 11d records this pixel (n, m) as one of the candidate pixels in the storage unit 12. The pixel selection unit 11d performs similar processing on the other deviation value maps.
[0059] The pixel selection unit 11d refers to all the pixels arranged in the n-th column. As a result, depending on the captured image, as shown in FIG. d A plurality of candidate pixels may be selected from a plurality of pixels arranged in the n-th column of the matrix.
[0060] In S105, the pixel selection unit 11d refers to the positions (coordinates) of a plurality of candidate pixels recorded in the storage unit 12. Then, a predetermined number or more (for example, three or more) of deviation value maps M d In the example shown in FIG. 11, a pixel (n, m) is selected as a common candidate pixel in the deviation value map M 300 M 700 M 1000Since the pixel (n, m) is a candidate pixel in the deviation value map M d The pixels selected as the candidate pixels in have a high probability of being able to calculate a highly accurate estimated depth d0. Therefore, such pixels are selected as common candidate pixels.
[0061] The pixel selection unit 11d selects this common candidate pixel as a depth display pixel (S106), as shown in Fig. 11. The pixel selection unit 11d also selects a predetermined number of pixels that sandwich the common candidate pixel in the pixel scanning direction (vertical direction in the example of Fig. 10) as depth display pixels (S106). For example, the pixel selection unit 11d selects two or three pixels that sandwich the common candidate pixel in the pixel scanning direction as depth display pixels.
[0062] The method of selecting common candidate pixels is not limited to the above example. For example, in contrast to the above example, the deviation value map M d Pixels selected as candidate pixels only in one or two deviation value maps M d A pixel selected as a candidate pixel only in the first place may be excluded from the candidate pixels, and all remaining candidate pixels may be selected as common candidate pixels.
[0063] [Pixel scanning direction] As described above, the pixel selection unit 11d scans the pixels in the vertical direction (pixel scanning direction) in the depth display pixel selection process. This is for the following reasons.
[0064] The first opening pattern B1 and the second opening pattern B2 illustrated in FIG. 4 are symmetrical with respect to a line Lc along the horizontal direction of the image. In this case, a relatively large difference occurs between the blur of the boundary line along the horizontal direction that appears in the captured image f1 and the blur of the boundary line along the horizontal direction that appears in the captured image f2. Therefore, a correct estimated depth d0 is calculated for pixels near the boundary line along the horizontal direction. In contrast, the difference between the blur of the boundary line along the vertical direction that appears in the captured image f1 and the blur of the boundary line along the vertical direction that appears in the captured image f2 becomes small. Therefore, the deviation value map M d When calculating w, the change in the deviation value w according to the reference depth becomes small for pixels corresponding to vertical boundary lines, and as a result, the difference between the actual depth and the estimated depth d0 becomes large for those pixels.
[0065] Therefore, in the depth display pixel selection process, the pixel selection unit 11d vertically scans the pixels to search for a pixel with a maximum deviation value w. In other words, the pixel selection unit 11d makes effective use of the disadvantage that it is difficult to calculate a correct estimated depth d0 for the boundary line in the vertical direction with the opening patterns B1 and B2, and scans the pixels vertically.
[0066] The pixel scanning direction is not limited to the vertical direction. For example, even if the aperture pattern of the coded aperture 14a is the pattern B1-B2 illustrated in Fig. 4, the pixel selector 11d may scan the pixels in the vertical direction to select candidate pixels, and then scan the pixels in another direction to select candidate pixels. In other words, the pixel selector 11d may scan the pixels in multiple directions.
[0067] Depending on the first opening pattern B1 and the second opening pattern B2, scanning may be performed in a direction different from the vertical direction (for example, the horizontal direction or a diagonal direction). Then, a pixel (candidate pixel) where the deviation value w is the maximum may be searched for.
[0068] For example, when the first aperture pattern B1 and the second aperture pattern B2 are symmetrical with respect to a vertical line, the pixel selection section 11d may search for candidate pixels by scanning pixels arranged in a direction perpendicular to this line (horizontal direction).
[0069] Also, for example, if the first opening pattern B1 and the second opening pattern B2 are symmetrical with respect to a line that is inclined with respect to the horizontal and vertical directions, the pixel selection unit 11d may scan the pixels in a direction perpendicular to this line to search for candidate pixels.
[0070] [Depth calculation using depth display pixels] The depth calculation unit 11c calculates a plurality of deviation value maps M d , and calculates the depth (the distance from the imaging system N to the subject) for the selected depth display pixel (S107). Specifically, as shown in FIG. 12, d 12, the depth calculation unit 11c searches for a reference depth that minimizes the deviation value w calculated for the depth display pixel. The depth calculation unit 11c then sets the reference depth found by the search as the estimated depth d0. The depth calculation unit 11c performs this search for all depth display pixels. In FIG. 12, the horizontal axis represents the reference depth, and the vertical axis represents the deviation value at the depth display pixel.
[0071] The depth calculation process is not limited to the example described here. In the above process, the depth calculation unit 11c calculates the deviation value w d The deviation value map M that minimizes d The reference depth corresponding to the depth display pixel is set as the estimated depth d0 of the depth display pixel. d A function showing the relationship between the pixel depth and the pixel depth may be calculated, and the minimum value of the function may be calculated. The depth at which the minimum value is obtained may be calculated as the estimated depth d0 of the pixel.
[0072] This process by the depth calculation unit 11c can be performed, for example, as follows. The depth calculation unit 11c calculates the reference depth and the deviation value w dThat is, the depth calculation unit 11c fits, for example, a cubic function to the relationship between the deviation value w d The following equation 3 is fitted to the point where d is the minimum and to multiple nearby points to find the coefficients a1·a2, a3, and a4. In equation 3, the function W(d) is a cubic function with coefficients a1·a2, a3, and a4.
[0073]
number
[0074] Then, the depth calculation unit 11c obtains the minimum of Equation 3. That is, the depth calculation unit 11c calculates the depth that minimizes the function W(d) by solving ΔW / Δd=0. Then, the depth calculation unit 11c sets the depth calculated in this way as the estimated depth d0. According to such processing, it is possible to increase the resolution regarding the depth without increasing the number of reference depths (the number of PSFs).
[0075] The depth calculation unit 11c determines whether or not the processes of S104 to S107 have been performed for all columns of the captured images f1 and f2 (S108). If there are any columns that have not yet been processed ("No" in S108), the depth calculation unit 11c returns to S104 and executes the processes of S104 to S107.
[0076] [Depth map display] When the processes of S104 to S107 are completed for all columns ("Yes" in S108), the map generating unit 11g displays the calculated estimated depth d0 on the depth map.
[0077] FIG. 13 is a diagram showing an example of a depth map displayed as a result of such processing. This diagram shows the result of processing for the checker pattern exemplified in FIG. 7. As shown in the lower diagram of FIG. 13, the estimated depths calculated for three pixels (depth display pixels) corresponding to the boundary between the white area and the black area are displayed on the depth map. In addition, the displayed estimated depth d0 is substantially unified to 1500 mm (the distance to the checker pattern, which is the subject). In contrast, the depths are not displayed for pixels that are not selected as depth display pixels. For this reason, only highly accurate depths are displayed on the depth map.
[0078] [Variations] The depth map generating device proposed in this disclosure is not limited to the example of generating device 10 described above.
[0079] In the generation device 10, in the depth display pixel selection process, the pixels in the deviation value map are scanned vertically to search for candidate pixels. The pixel scanning direction is not limited to the vertical direction and may be determined according to the first opening pattern and the second opening pattern.
[0080] Fig. 14 is a diagram showing a modified example of the aperture pattern. The first aperture pattern B3 and the second aperture pattern B4 shown in this figure have a rectangular light-shielding region R3, a circular light-transmitting region R1, and a circular light-shielding region R2, similar to the example shown in Fig. 4. Unlike the example in Fig. 4, the light-shielding region R2 of the first aperture pattern B3 and the light-shielding region R2 of the second aperture pattern B4 are symmetrical with respect to a line Le that is inclined with respect to both the vertical and horizontal directions of the image.
[0081] When a checkered pattern as exemplified in FIG. 15 is photographed through such opening patterns B3 and B4, there is a large difference between the blur of the boundary line extending in the direction along the line Le (see FIG. 14) in the captured image f1 and the blur of the boundary line extending in the direction along the line Le (see FIG. 14) in the captured image f2. Therefore, a highly accurate depth can be calculated for the position of this boundary line. In contrast, the difference in the blur of the boundary line Lb (see FIG. 15) extending in a direction perpendicular to the line Le is small. In this case, the pixel selection unit 11d may scan the pixels in each deviation value map in a direction S1 (see FIG. 15) perpendicular to the line Le, and search for a pixel (i.e., a candidate pixel) where the deviation value is maximized (S104).
[0082] The subsequent processing is the same as that of the above-mentioned generating device 10. That is, the pixel selection unit 11d selects a predetermined number or more of deviation value maps M d In step S108, the map generating unit 11g selects pixels that are candidate pixels in the pixel scanning direction S1 as common candidate pixels (S105). Then, the common candidate pixel and a predetermined number of pixels sandwiching the common candidate pixel in the pixel scanning direction S1 are selected as depth display pixels (S106). The depth calculating unit 11c calculates an estimated depth for the selected depth display pixel (S107). If it is determined in step S108 that the processes in steps S104 to S107 have been performed for all pixel rows along the pixel scanning direction S1, the map generating unit 11g displays the calculated estimated depth as a depth map (S109).
[0083] [summary] (1) The depth map generating device 10 includes a depth calculation unit 11c that calculates depths for a plurality of pixels, a pixel selection unit 11d that selects depth display pixels that are pixels that indicate depths in the depth map, and a storage unit 12 that stores a PSF that corresponds to a predefined reference depth and an aperture pattern of a coded aperture. The pixel selection unit 11d calculates a deviation value w that indicates the degree of deviation between the reference depth and the actual depth for each pixel based on a reference depth restored image reconstructed from the captured images f1 and f2 using the PSF and the captured images f1 and f2, and selects a depth display pixel based on a change in the deviation value of pixels aligned along the pixel scanning direction. According to the depth map generating device 10 of (1), a depth display pixel is selected based on a change in deviation value w of pixels aligned along the pixel scanning direction, so that the accuracy of the depth displayed on the depth map can be improved. For example, the accuracy of the depth displayed on the depth map can be improved compared to a mode in which a pixel having a large difference between the maximum and minimum deviation values w (for example, Δwa·Δwb in FIG. 8) is selected as a depth display pixel.
[0084] (2) In the generating device 10 of (1), the pixel selector 11d calculates a first deviation value representing the degree of deviation between the first reference depth and the actual depth for each pixel based on the first reference depth restored image reconstructed from the captured images f1 and f2 using the first PSF and the captured images f1 and f2. Then, the pixel selector 11d selects one or more pixels as first candidate pixels based on the change in the first deviation value of the pixels aligned along the pixel scanning direction. The pixel selector 11d also calculates a second deviation value representing the degree of deviation between the second reference depth and the actual depth for each pixel based on the second reference depth restored image reconstructed from the captured images f1 and f2 using the second PSF and the captured images f1 and f2. Then, the pixel selector 11d selects one or more pixels as second candidate pixels based on the change in the second deviation value of the pixels aligned along the pixel scanning direction. If the first candidate pixel and the second candidate pixel satisfy a predetermined condition, the pixel selection unit 11d selects the first candidate pixel and the second candidate pixel as depth display pixels.
[0085] According to the generating device 10 of (2), a depth display pixel is selected using two candidate pixels calculated from two reference depth restored images. Therefore, for example, the accuracy of the depth displayed in the depth map can be improved compared to a case where a depth display pixel is selected using only candidate pixels calculated from one reference depth restored image.
[0086] (3) In the generating device 10 of (1) or (2), the candidate pixel is a pixel that gives the maximum deviation value of the pixels aligned along the pixel scanning direction. The deviation value is large for pixels for which the depth is calculated with high accuracy. The generating device 10 of (3) can utilize this to select the candidate pixel.
[0087] (4) In the generating device 10 of (2), the pixel selector 11d selects a plurality of candidate pixels including a first candidate pixel and a second candidate pixel from a plurality of reference depth restored images. When the positions (coordinates) of a predetermined number of candidate pixels included in the plurality of candidate pixels match, the pixel selector 11d selects the candidate pixel with the matching position as a depth display pixel. This can further improve the accuracy of the depth displayed on the depth map.
[0088] (5) In the generating device 10 of (1) to (4), the coded aperture 14a includes a first aperture pattern B1·B3 and a second aperture pattern B2·B4. The depth map generating device 10 generates a depth map from a first captured image f1 captured through the first aperture pattern B1·B3 and a second captured image f2 captured through the second aperture pattern B2·B4. This can improve the accuracy of the depth displayed in the depth map compared to when the number of aperture patterns is one.
[0089] (6) In the generating device 10 of (1) to (5), the pixel scanning direction is determined according to the first opening patterns B1 and B3 and the second opening patterns B2 and B4. This makes it possible to calculate the depth by effectively utilizing the disadvantage that it is difficult to calculate the correct depth depending on the relationship between the opening pattern and the pixel scanning direction.
[0090] In the generating apparatus 10 of (7)(6), the first aperture patterns B1·B3 and the second aperture patterns B2·B4 are symmetrical to each other with respect to the straight line Lc·Le. The pixel scanning direction intersects with the straight line Lc·Le. [Explanation of symbols]
[0091] 10: depth map generating device, 11: control unit, 11a: image acquisition unit, 11b: aperture control unit, 11c: depth calculation unit, 11d: pixel selection unit, 11g: map generating unit, 12: memory unit, 13: image sensor, 14: liquid crystal panel, 14a: coded aperture, 15: lens, 16: input unit, B1, B2, B3, B4: aperture pattern, R1: light transmitting region, R2: light blocking region, R3: light blocking region.
Claims
1. A depth map generating device for generating a depth map including a plurality of pixels each representing a depth from an image captured through a coded aperture, the device comprising: a depth calculation unit that calculates a depth for the plurality of pixels; a pixel selection unit that selects depth display pixels that are pixels that indicate depth in the depth map; A storage unit that stores a PSF corresponding to a predefined reference depth and corresponding to an aperture pattern of the coded aperture; Including, The pixel selection unit calculates a deviation value representing a degree of deviation between the reference depth and an actual depth for each pixel based on the captured image and a reference depth restored image reconstructed from the captured image using the PSF, and selects the depth display pixel based on a change in the deviation value of pixels aligned in a predetermined direction. Depth map generator.
2. the storage unit stores a plurality of PSFs each corresponding to a plurality of reference depths that are discretely set and corresponding to an aperture pattern of the coded aperture; The plurality of reference depths include at least a first reference depth and a second reference depth, the plurality of PSFs include a first PSF corresponding to a first reference depth and a second PSF corresponding to a second reference depth; The pixel selection unit calculating, for each pixel, a first deviation value representing a degree of deviation between the first reference depth and an actual depth based on the captured image and a first reference depth restored image reconstructed from the captured image using the first PSF; selecting one or more pixels as first candidate pixels based on a change in the first deviation value of pixels aligned along a predetermined direction; calculating, for each pixel, a second deviation value representing a degree of deviation between the second reference depth and an actual depth based on the captured image and a second reference depth restored image reconstructed from the captured image using the second PSF; selecting one or more pixels as second candidate pixels based on a change in the second deviation value of the pixels aligned along the predetermined direction; If the first candidate pixel and the second candidate pixel satisfy a predetermined condition, the first candidate pixel and the second candidate pixel are selected as the depth display pixel. The apparatus for generating a depth map according to claim 1 .
3. The candidate pixel is a pixel that gives a maximum of the deviation value of the pixels aligned in the predetermined direction. The apparatus for generating a depth map according to claim 1 .
4. the pixel selection unit selects a plurality of candidate pixels including the first candidate pixel and the second candidate pixel from a plurality of reference depth restored images, When positions of a predetermined number of candidate pixels included in the plurality of candidate pixels match, the pixel selection unit selects the matching candidate pixels as the depth display pixel. The apparatus for generating a depth map according to claim 2 .
5. the coded aperture includes a first aperture pattern and a second aperture pattern; The depth map generating device generates the depth map from a first captured image captured through the first aperture pattern and a second captured image captured through the second aperture pattern. The depth map generating apparatus according to claim 1 .
6. The predetermined direction is a direction determined according to the first opening pattern and the second opening pattern. The depth map generating apparatus according to claim 1 .
7. the first opening pattern and the second opening pattern are symmetrical with respect to a line, The predetermined direction is a direction intersecting the straight line. A depth map generating apparatus according to claim 6.
8. A method for generating a depth map, comprising: generating a depth map including a plurality of pixels each representing a depth from an image captured through a coded aperture, the method comprising: a depth calculation step of calculating a depth for the plurality of pixels; a pixel selection step of selecting depth display pixels in the depth map, the depth display pixels being pixels that indicate depth; Obtaining a PSF corresponding to a predefined reference depth and corresponding to an aperture pattern of the coded aperture from a storage unit; Including, In the pixel selection step, a deviation value representing a degree of deviation between the reference depth and an actual depth is calculated for each pixel based on the captured image and a reference depth restored image reconstructed from the captured image using the PSF, and the depth display pixel is selected based on a change in the deviation value of pixels aligned in a predetermined direction. How the depth map is generated.
9. A program for causing a computer to function as a depth map generating device for generating a depth map including a plurality of pixels each representing a depth from an image captured through a coded aperture, the program comprising: a depth calculation means for calculating a depth for the plurality of pixels; a pixel selection means for selecting depth display pixels in the depth map, the depth display pixels being pixels that indicate depth; and A means for acquiring a PSF corresponding to a predefined reference depth and corresponding to an aperture pattern of the coded aperture from a storage unit. The computer functions as The pixel selection means calculates a deviation value representing a degree of deviation between the reference depth and an actual depth for each pixel based on the captured image and a reference depth restored image reconstructed from the captured image using the PSF, and selects the depth display pixel based on a change in the deviation value of pixels aligned in a predetermined direction. program.
