Imaging device equipped with a liquid crystal panel

Abstract

This improves the practicality of depth estimation of subjects using encoded imaging. [Solution] An imaging device is provided comprising an encoding imaging system and one or more processors that control the encoding imaging system, wherein the encoding imaging system includes an encoding mask, the encoding mask includes a liquid crystal panel, and the one or more processors perform the following: a first encoding imaging process in which the encoding imaging system controls the system to encode and image a subject at a first exposure level to obtain a first image; an exposure determination process in which the system determines a second exposure level based on the brightness values ​​of a plurality of pixels constituting the first image; a second encoding imaging process in which the system controls the system to encode and image a subject at a second exposure level to obtain a second image; and a depth estimation process in which the system estimates the distances from the encoding imaging system to a plurality of positions on the subject, i.e., depth or length, based on information obtained by applying a decoding process to the second image.

Description

Technical Field

[0001] The present invention relates to an imaging device, and more particularly to an imaging device provided with a liquid crystal panel.

Background Art

[0002] In the field of coded imaging, a technique called DFD (Depth From Defocus) is known. The DFD technique is a technique for estimating the distance from the optical system of an imaging device to a subject, that is, the depth or depth of the subject, based on the degree of blurring of the edges captured in the image obtained by imaging.

[0003] The DFD technique is described in, for example, Non-Patent Document 1. In the DFD technique, coded imaging is performed in which a mask called a coded aperture is arranged in the light incident region of the optical system to image the subject. Next, the captured image obtained by the coded imaging is subjected to a decoding process based on the point spread function unique to the mask, and the depth of the subject is estimated. The point spread function is generally called a PSF (Point Spread Function), and is also referred to as a blur function, a blur spread function, a point image distribution function, and the like.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] DFD technology is still under development, and there is much room for improvement in terms of practicality. For these reasons, there is a need for more practical DFD technology. [Means for solving the problem]

[0006] The following is a summary of some of the representative inventions disclosed in this application.

[0007] One embodiment is an imaging device comprising an encoding imaging system and one or more processors that control the encoding imaging system, wherein the encoding imaging system includes an encoding mask, the encoding mask includes a liquid crystal panel, and the one or more processors perform the following: a first encoding imaging process for controlling the encoding imaging system to encode and image a subject at a first exposure level and obtaining a first image; an exposure determination process for determining a second exposure level based on the brightness values ​​of a plurality of pixels constituting the obtained first image; a second encoding imaging process for controlling the encoding imaging system to encode and image the subject at the determined second exposure level and obtaining a second image; and a depth estimation process for estimating the depth from the encoding imaging system to a plurality of positions on the subject based on information obtained by applying a decoding process to the obtained second image.

[0008] Furthermore, one embodiment is a subject depth estimation method in which one or more processors perform a first encoding imaging process to obtain a first image by controlling an encoding imaging system to encode and image a subject at a first exposure level; an exposure determination process to determine a second exposure level based on the brightness values ​​of a plurality of pixels constituting the obtained first image; a second encoding imaging process to obtain a second image by controlling the encoding imaging system to encode and image the subject at the determined second exposure level; and a depth estimation process to estimate the depth of each of a plurality of positions on the subject from the encoding imaging system based on information obtained by decoding the obtained second image.

[0009] Furthermore, one embodiment is a program that causes one or more processors to execute: a first encoding imaging process that controls an encoding imaging system to encode and image a subject at a first exposure level and obtain a first image; an exposure determination process that determines a second exposure level based on the brightness values ​​of a plurality of pixels constituting the obtained first image; a second encoding imaging process that controls the encoding imaging system to encode and image the subject at the determined second exposure level and obtain a second image; and a depth estimation process that estimates the depth from the encoding imaging system to a plurality of positions on the subject based on information obtained by decoding the obtained second image. [Brief explanation of the drawing]

[0010] [Figure 1] This diagram schematically shows an example of the installation of the imaging device according to Embodiment 1. [Figure 2] This figure shows an example of the configuration of the imaging device according to Embodiment 1. [Figure 3] This figure shows an example of the hardware configuration of the arithmetic control processing unit according to Embodiment 1. [Figure 4] This figure shows an example of the functional block configuration of the arithmetic control processing unit according to Embodiment 1. [Figure 5] This figure shows an example of a histogram generated by the imaging device according to Embodiment 1. [Figure 6] This figure shows an example of a histogram generated by the imaging device according to Embodiment 1. [Figure 7] This figure shows an example of a table used in the imaging device according to Embodiment 1. [Figure 8] This is a flowchart showing an example of the operation flow in the imaging device according to Embodiment 1. [Figure 9] This figure illustrates a modified example 1 of Embodiment 1. [Figure 10] This figure shows an example of a table relating to a modified example 2 of Embodiment 1. [Figure 11]It is a diagram showing an example of a table according to Modification Example 2 of Embodiment 1. [Figure 12] It is a diagram showing an example of a table according to Modification Example 2 of Embodiment 1. [Figure 13] It is a diagram showing the flow of processing in an imaging device according to Modification Example 3 of Embodiment 1. [Figure 14] It is a diagram showing the flow of processing in an imaging device according to Modification Example 4 of Embodiment 1. [Figure 15A] It is a diagram schematically showing an imaging image in which there is no area where the luminance value is saturated. [Figure 15B] It is a diagram schematically showing an imaging image in which there is an area where the luminance value is saturated.

Mode for Carrying Out the Invention

[0011] <Background of the Study by the Present Inventors> Before explaining the embodiments of the present invention, the basic content of the DFD (Depth From Defocus) technology, which is a spatial recognition technology, and the problems discovered by the present inventors will be explained.

[0012] The blurring state of the subject in the imaging image generally depends on the point spread function determined by the optical system of the imaging device, the shape of the light incident area of the optical system, etc. When a mask for partially shielding light is installed in the light incident area of the optical system, the point spread function is determined for each mask. Imaging a subject with an imaging device equipped with a mask is called coded imaging. When the subject is coded imaged, a blurred image based on the point spread function unique to the used mask is acquired as the imaging image.

[0013] When a decoding process of performing inverse convolution based on the point spread function unique to the used mask is performed on this blurred image, which is the imaging image, a decoded image with improved blurring and the depth information of the object corresponding to each position of the subject included in the decoded image can be obtained.

[0014] On the one hand, as described above, when the inventors perform a decoding process on the captured image obtained by encoding and capturing a subject with an imaging device to estimate the depth at each position of the subject, they have found the following problems.

[0015] The inventors have found through research and development of DFD technology that the accuracy of subject depth estimation is related to the band of the luminance value of each pixel in the captured image obtained by encoding and capturing. In particular, it has been confirmed that the presence of a region where the luminance value is saturated in the captured image, that is, a region where so-called "blown out highlights" occur, significantly reduces the accuracy of subject depth estimation.

[0016] FIG. 15A is a diagram schematically showing a captured image without a region where the luminance value is saturated. As shown in FIG. 15A, in the case of the captured image Pa without a region where the luminance value is saturated, there is no lack of information on the degree of blur of the image necessary for subject depth estimation. Therefore, when a decoding process is performed on this captured image Pa to estimate the subject depth, appropriate estimation can be performed.

[0017] FIG. 15B is a diagram schematically showing a captured image with a region where the luminance value is saturated. As shown in FIG. 15B, in the case of the captured image Pw with a region where the luminance value is saturated, there is a lack of information on the degree of blur of the image necessary for subject depth estimation. Therefore, when a decoding process is performed on this captured image Pw to estimate the subject depth, appropriate estimation cannot be performed.

[0018] Due to the above circumstances, in the DFD technology, a technology that can stably estimate the depth of a subject with high accuracy regardless of the brightness of the subject is desired.

[0019] In view of the above circumstances, the present inventors have devised the present invention as a result of diligent study. Embodiments of the present invention will be described below. The embodiments described below are examples for carrying out the present invention and do not limit the technical scope of the present invention. In the following embodiments, components having the same function are denoted by the same reference numerals, and repeated descriptions thereof will be omitted unless particularly necessary.

[0020] (Embodiment 1) <Overview of the imaging device> The imaging device according to Embodiment 1 of this application will now be described. The imaging device according to Embodiment 1 comprises an encoding imaging system and one or more processors that control the encoding imaging system. The one or more processors perform the following processes.

[0021] (1) A first encoding imaging process that controls the encoding imaging system to encode and image the subject at a first exposure level and obtains a first image. (2) Exposure determination process that determines a second exposure level based on the brightness values ​​of multiple pixels that constitute the first captured image, (3) A second encoding imaging process that controls the encoding imaging system to encode and image the subject at a second exposure level to obtain a second image, (4) Depth estimation process that estimates the depth of each of multiple positions on the subject from the encoded imaging system based on the information obtained by applying a decoding process to the second captured image. The details of the imaging device according to Embodiment 1 will be described below.

[0022] <Hardware configuration of the imaging device> Figure 1 is a schematic diagram showing an example of the installation of the imaging device 1 according to Embodiment 1. In the figure, the z direction is the forward direction of the automobile 100.

[0023] As shown in Figure 1, the imaging device 1 is installed on a vehicle, which is an automobile 100. The imaging device 1 is positioned to encode and capture a subject 90 located in front of the automobile 100.

[0024] Figure 2 shows an example of the configuration of the imaging device 1 according to Embodiment 1. As shown in Figure 2, the imaging device 1 includes an encoding imaging system 2, an arithmetic control processing unit 3, and an imaging system control unit 11.

[0025] The imaging system control device 11 is electrically or communicatively connected to the encoding imaging system 2 and the arithmetic control processing unit 3. Based on the control signal CL from the arithmetic control processing unit 3, the imaging system control device 11 controls the components of the encoding imaging system 2 to set imaging conditions for encoding imaging and to perform encoding imaging under the set imaging conditions.

[0026] The encoding imaging system 2 is installed as part of the imaging device 1, for example, in the front of the interior of a car 100. The encoding imaging system 2 includes a mask 21, an optical system 22, an image sensor 23, and an aperture 24.

[0027] The mask 21 has a specific geometric aperture pattern and functions as a filter for light that enters the optical system 22 from the subject 90 and reaches the image sensor 23. The mask 21 may be placed between the optical system 22 and the image sensor 23. The mask 21 is also called an encoded aperture. The mask 21 includes, for example, a liquid crystal panel, in which the light-transmitting and light-blocking regions of the liquid crystal panel are controlled to form a desired aperture pattern.

[0028] The optical system 22 focuses the light incident from the subject 90 onto the light-receiving surface 23a of the image sensor 23 to form an image. The optical system 22 is, for example, a lens. The lens may be a single lens or a compound lens, and may be a fixed-focal-length lens or a zoom lens.

[0029] The image sensor 23 has a light-receiving surface 23a, which is composed of a plurality of photoelectric conversion elements arranged in two dimensions. The image sensor 23 converts the intensity of the received light L into an electrical signal corresponding to the amount of light received within a certain period of time for each photoelectric conversion element on the light-receiving surface 23a, and generates image data representing the captured image based on the electrical signals of each photoelectric conversion element. The image sensor 23 outputs the generated image data of the captured image to the arithmetic control processing unit 3. Alternatively, the image sensor 23 may output the photoelectrically converted electrical signals to the arithmetic control processing unit 3, so that the arithmetic control processing unit 3 can generate image data of the captured image based on those electrical signals. The image sensor 23 is also called an image sensor.

[0030] The aperture 24 has a variable-sized opening. By changing the size of this opening, the aperture 24 controls the amount of light L incident from the subject 90 and received by the light-receiving surface 23a of the image sensor 23, that is, the amount of light received by the light-receiving surface 23a when performing encoded imaging. The aperture 24 is positioned between the subject 90 and the mask 21, between the mask 21 and the optical system 22, or between the optical system 22 and the image sensor 23. Alternatively, the aperture 24 may also have the function of a mask 21. In this case, the aperture 24 with the mask function is positioned between the image sensor 23 and the optical system 22.

[0031] The encoded imaging system 2 has a point spreading function unique to that encoded imaging system 2. The point spreading function is a function that determines how point images on the subject appear blurred in the captured image, and is determined by the combination of the mask 21, the optical system 22, the image sensor 23, and the aperture 24.

[0032] The arithmetic control processing unit 3 sends a control signal CL to the imaging system control unit 11, causing the imaging system control unit 11 to control the encoding imaging system 2. Specifically, for example, the arithmetic control processing unit 3 causes the imaging system control unit 11 to set imaging conditions for encoding imaging, or to perform encoding imaging under the set imaging conditions. The imaging conditions for encoding imaging include, for example, the aperture pattern of the mask 21 and the exposure level.

[0033] The exposure level is determined by the combination of the aperture value of f / 24 and the shutter speed. The aperture value is an index value that represents the size of the opening at f / 24, or the amount of light passing through this opening, and is also called the f-number. Shutter speed generally refers to the time the shutter is open. In the case of an electronic shutter, the shutter speed is, for example, the storage time of the electrical signal converted photoelectrically by the image sensor 23. In this embodiment, an electronic shutter is used as the shutter. However, a physical shutter may be provided in the encoding imaging system 2, and the shutter speed may be the time this physical shutter is open.

[0034] Note that there are usually multiple combinations of aperture and shutter speed that produce the same exposure level. When changing the exposure level, you can consider three methods: fixing the aperture and changing the shutter speed, fixing the shutter speed and changing the aperture, or changing both the aperture and shutter speed. Any of these methods is acceptable for changing the exposure level.

[0035] The arithmetic control processing unit 3 instructs the imaging system control unit 11 to repeatedly perform encoded imaging for the required period. Based on the distribution, bandwidth, or level of the brightness values ​​of each pixel in the image obtained by encoded imaging, the arithmetic control processing unit 3 controls the exposure level during encoded imaging so that the region where the brightness values ​​in the image are saturated is reduced. The arithmetic control processing unit 3 performs a decoding process on the image obtained by encoded imaging P. By performing this decoding process, the arithmetic control processing unit 3 obtains a decoded image with improved blur of the subject 90 and depth estimates from the optical system at each position of the subject 90. The arithmetic control processing unit 3 generates a depth map DM by associating the depth estimates with the decoded image and outputs the depth map DM to an external device 4, etc.

[0036] Figure 3 shows an example of the hardware configuration of the arithmetic control processing unit 3 according to Embodiment 1. In this embodiment, the arithmetic control processing unit 3 is, for example, a computer. The arithmetic control processing unit 3 has a processor 301, memory 302, storage 303, interface 304, and communication bus 305. The processor 301, memory 302, storage 303, and interface 304 are connected to each other so as to be able to communicate with each other via the communication bus 305. At least a part of the arithmetic control processing unit 3 may be composed of semiconductor devices such as FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit).

[0037] The processor 301 may include, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an MCU (Micro Controller Unit), or a GPU (Graphics Processing Unit). The processor 301 performs various calculations and processes. The processor 301 may consist of one CPU, MCU, GPU, etc., or it may consist of two or more.

[0038] Memory 302 is, for example, a semiconductor memory device. Memory 302 temporarily stores data used for processing when the processor 301 performs some kind of processing.

[0039] Storage 303 can be, for example, an SSD (Solid State Drive), an HDD (Hard Disk Drive), a USB memory stick, or an eMMC (Embedded Multi Media Card). Storage 303 stores the program PR. The processor 301 reads the program PR stored in storage 303, expands it into memory 302, and executes it, thereby allowing the arithmetic control processing unit 3 to function as various functional blocks.

[0040] Interface 304 receives electrical signals or image data representing the captured image P from the image sensor 23, and outputs depth information of the subject 90, such as a depth map DM, to the outside.

[0041] <Functional block configuration of the arithmetic control unit> Figure 4 shows an example of the functional block configuration of the arithmetic control processing unit 3 according to Embodiment 1. As described above, this functional block is realized by the processor 301 executing the program PR. As shown in Figure 4, the arithmetic control processing unit 3 has as functional blocks an overall control unit 31, an encoded imaging control unit 32, a brightness value histogram generation unit 33, a feature quantity calculation unit 34, an exposure level determination unit 35, a table storage unit 36, a decoding processing unit 37, and a depth map generation unit 38.

[0042] The central control unit 31 controls each part so that the encoding and imaging of the subject 90, reception of the captured image P, generation of a histogram H of luminance values ​​in the captured image P, calculation of a feature quantity F related to the distribution of luminance values, determination of the exposure level E, decoding of the captured image P, estimation of the depth of the subject 90, and generation of a depth map DM are performed under appropriate conditions and in the correct order.

[0043] The encoding imaging control unit 32 controls the imaging system control unit 11 so that encoding imaging is performed with the set exposure level E and mask pattern MP.

[0044] The luminance value histogram generation unit 33 generates a histogram H of the luminance values ​​of each pixel in the captured image P, based on the captured image P obtained by encoded imaging.

[0045] The feature calculation unit 34 calculates feature quantities related to the distribution, bandwidth, and level of brightness values ​​of each pixel in the captured image P, such as feature quantities F related to the brightness level, based on the generated histogram H. In this embodiment, feature quantity F is defined as the number of maximum brightness pixels (hereinafter also referred to as the maximum brightness pixel count) that have a brightness value equal to the maximum brightness value corresponding to the highest tone in the brightness gradation, reflecting the extent of the region where the brightness value in the captured image is saturated. However, pixels that happen to have the highest brightness due to noise or the like are not counted as maximum brightness pixels as much as possible, and the pixels to be counted are narrowed down by a predetermined method.

[0046] Here, we will explain how to count the number of pixels with maximum brightness as a feature. Figure 5 shows an example of a histogram generated by the imaging device 1 according to Embodiment 1. Figure 5 shows an example of a histogram of brightness values ​​for all pixels in an captured image. In the histogram H1 shown in Figure 5, the vertical axis represents the brightness value B of the pixels, and the horizontal axis represents the number of pixels T.

[0047] In this embodiment, as shown in Figure 5, the upper brightness value Bu is obtained by adding three times the standard deviation σ of the brightness values ​​of all pixels to the average value Ba of the brightness values ​​of all pixels that make up the captured image P. Furthermore, among all pixels, the number of pixels with brightness values ​​less than or equal to the upper brightness value Bu that have a brightness value equal to the maximum brightness value Bmax, which corresponds to the highest tone in the brightness gradation, is counted as the maximum brightness pixel count Tz. In this embodiment, this maximum brightness pixel count Tz is the feature quantity F.

[0048] Here, the pixels to be counted for the maximum brightness pixel count Tz are limited to those with a brightness value less than or equal to the upper brightness value Bu. This is to prevent pixels that have reached the maximum brightness value Bmax due to external noise, non-operation of the photoelectric conversion element, etc., from being counted as the maximum brightness pixel count Tz. Assuming that the brightness values ​​of each pixel constituting the captured image P follow a statistically normal distribution, pixels in the range of mean ± standard deviation σ × 3, that is, pixels in the range of upper brightness value Bu or less and lower brightness value Bd or greater as shown in Figure 5, correspond to 99.7% of all pixels in the captured image P. Therefore, this method makes it possible to count the maximum brightness pixel count Tz without compromising the trend of the brightness value distribution and eliminating the effects of external noise, malfunction of the photoelectric conversion element, etc.

[0049] In the example of histogram H1 shown in Figure 5, there are no pixels with a brightness value equal to the maximum brightness value Bmax below the upper brightness value Bu; therefore, the number of pixels with the highest brightness Tz1 in this example is zero (0). In this example, it can be considered that there are no pixel regions where the brightness value is saturated.

[0050] Figure 6 shows an example of a histogram generated by the imaging device 1 according to Embodiment 1. Figure 6 shows an example of a histogram of luminance values ​​for all pixels in the captured image P. In the example of histogram H2 shown in Figure 6, the luminance values ​​are generally higher compared to the example of histogram H1. In the example of histogram H2, there are pixels with luminance values ​​equal to the maximum luminance value Bmax that are below the upper luminance value Bu, and the number of pixels with the highest luminance Tz2 in this example is Tz2 > 0. That is, in this example, it can be considered that there are pixel regions where the luminance value is saturated.

[0051] The exposure level determination unit 35 determines the exposure level in encoded imaging. In this embodiment, the exposure level determination unit 35 refers to the table TB described later and determines a new exposure level E in encoded imaging based on the calculated feature quantity F, i.e., the maximum brightness pixel count Tz. Alternatively, the exposure level determination unit 35 may determine the new exposure level E using a function that outputs an exposure level E when the feature quantity F is input.

[0052] The table storage unit 36 stores a table TB that associates the feature amount F with the exposure level determination method. In the present embodiment, the table TB associates the maximum luminance pixel count Tz with the determination method of the exposure level E.

[0053] FIG. 7 is a diagram showing an example of a table used in the imaging device 1 according to Embodiment 1. FIG. 7 shows an example of a table that associates a feature amount with an exposure level determination method. In the present embodiment, as shown in FIG. 7, the table TB represents a table in which the maximum luminance pixel count Tz as the feature amount F and the determination method of the exposure level E to be applied according to the maximum luminance pixel count Tz are associated. The exposure level determination unit 35 refers to this table TB, specifies the exposure level determination method according to the maximum luminance pixel count Tz calculated by the feature amount calculation unit 34, and determines the exposure level E using the specified exposure level determination method.

[0054] In the example of the table TB shown in FIG. 7, for example, when the maximum luminance pixel count Tz = 0, the new exposure level E is determined to be the same exposure level as the initially set exposure level Ec. When the maximum luminance pixel count Tz satisfies 0 < Tz ≤ T1, the exposure level Ec - 1Le obtained by subtracting 1Le corresponding to one level from the initially set exposure level Ec is determined as the new exposure level E. When the maximum luminance pixel count Tz satisfies T1 < Tz ≤ T2, the exposure level Ec - 2Le obtained by subtracting 2Le corresponding to two levels from the initially set exposure level Ec is determined as the new exposure level E. Further, when the maximum luminance pixel count Tz satisfies Tn - 1 < Tz ≤ Tn, the exposure level Ec - nLe obtained by subtracting nLe corresponding to n levels from the initially set exposure level Ec is determined as the new exposure level E.

[0055] Note that, in the example of the table TB shown in FIG. 7, the feature amount F is associated with the determination method of the new exposure level E, but the table TB may directly associate the feature amount F with the new exposure level E.

[0056] According to this method of determining the exposure level E based on the maximum brightness pixel count Tz, the maximum brightness pixel count Tz can first be determined as an indicator of the size of the pixel region in the captured image P where the brightness value is saturated, i.e., the pixel region where the image is "blown out." The larger the pixel region where the brightness value is saturated, the lower the exposure level can be, and the distribution of brightness values ​​for each pixel in the captured image P can be made to fall within an appropriate range that does not include the saturated region as much as possible.

[0057] The decoding processing unit 37 receives the captured image P obtained by encoded imaging from the image sensor 23 in response to control from the overall control unit 31, and performs decoding on the captured image P. The decoding of the captured image P is a deconvolution process based on a point spreading function determined by the aperture pattern of the mask 21, the optical system 22, the image sensor 23, the aperture 24, etc. When the decoding of the captured image P is performed, a decoded image M, which is an image of the subject 90 with improved blur, and a depth estimate value dj of the subject 90 from the optical system 22 are obtained. In this specification, this depth estimate value dj represents the set of depth estimate values ​​at each position of the subject 90.

[0058] The depth map generation unit 38 generates a depth map DM in which the depth estimate value dj at each position of the subject 90 included in the decoded image M is associated with the control unit 31, in response to control from the control unit 31. The depth map generation unit 38 also outputs the generated depth map DM to an external device 4 or the like, in response to control from the control unit 31.

[0059] <Processing flow in imaging device> The following describes the operation flow of the imaging device 1. Figure 8 is a flowchart showing an example of the operation flow in the imaging device 1 according to Embodiment 1.

[0060] As shown in Figure 8, in step S1, encoding imaging is performed at a first exposure level. Specifically, the encoding imaging control unit 32 sends a control signal to the imaging system control unit 11, and the imaging system control unit 11 controls the encoding imaging system 2 based on the received control signal. The encoding imaging system 2, under the control of the imaging system control unit 11, performs encoding imaging of the subject 90 at the initially set first exposure level E1. The first exposure level E1 is an exposure level that, empirically or statistically, often does not produce a saturation region of brightness values ​​in the captured image P and does not result in loss of information regarding the degree of blur. The information representing the captured image P1 obtained by this encoding imaging is sent to the brightness value histogram generation unit 33 via the overall control unit 31 or directly. When the processing of step S1 is completed, the processing steps proceed to step S2.

[0061] In step S2, a histogram of brightness values ​​in the captured image is generated. Specifically, the brightness value histogram generation unit 33 generates a histogram H of brightness values ​​for all pixels in the captured image P1 based on the captured image P1 obtained in step S1. The information representing the generated histogram H is sent to the feature calculation unit 34. Once the processing in step S2 is completed, the processing steps proceed to step S3.

[0062] In step S3, features related to the distribution of luminance values ​​are calculated. Specifically, the feature calculation unit 34 calculates features F based on the histogram H generated in step S2. In this embodiment, the feature calculation unit 34 uses the luminance value obtained by the average value of luminance values ​​Ba + the standard deviation of luminance values ​​σ × 3 as the upper luminance value Bu, and calculates the maximum luminance pixel count Tz as the number of pixels with the maximum luminance value Bmax among pixels having a luminance value less than or equal to the upper luminance value Bu. The information representing the calculated maximum luminance pixel count Tz is sent to the exposure level determination unit 35. Once the processing in step S3 is completed, the processing steps proceed to step S4.

[0063] In step S4, a second exposure level is determined based on feature quantities. Specifically, the exposure level determination unit 35 determines a method for determining the second exposure level E2 by referring to the table TB stored in the table storage unit 36, based on the maximum brightness pixel count Tz calculated in step S3. The exposure level determination unit 35 determines the second exposure level E2 using the determined determination method. The second exposure level E2 is an exposure level adjusted relative to the first exposure level E1 so that there are no regions where the brightness value is saturated in the captured image P obtained by encoded imaging. The information representing the determined second exposure level E2 is sent to the encoded imaging control unit 32. Once the processing in step S4 is completed, the processing steps proceed to step S5.

[0064] In step S5, encoded imaging is performed at a second exposure level. Specifically, the encoded imaging control unit 32 sends a control signal to the imaging system control unit 11. The imaging system control unit 11 controls the encoded imaging system 2 based on the received control signal. The encoded imaging system 2, under the control of the imaging system control unit 11, performs encoded imaging of the subject 90 at the second exposure level E2 determined in step S4. The information representing the captured image P2 obtained by this encoded imaging is sent to the decoding processing unit 37 via the overall control unit 31 or directly. Once the processing in step S5 is completed, the processing steps proceed to step S6.

[0065] In step S6, depth estimation of the subject is performed based on the decoding of the captured image. Specifically, the decoding processing unit 37 performs decoding on the captured image P2 obtained in step S5 to obtain a decoded image M with improved blur and depth estimates dj for each position of the subject 90. The information representing the decoded image M and the depth estimates dj of the subject 90 is sent to the depth map generation unit 38. Once the processing in step S6 is completed, the processing steps proceed to step S7.

[0066] In step S7, a depth map is generated. Specifically, the depth map generation unit 38 generates a depth map DM based on the decoded image M obtained in step S6 and the depth estimate dj of the subject 90. Once the processing in step S7 is complete, the processing steps proceed to step S8.

[0067] In step S8, the depth map is output. Specifically, the depth map generation unit 38 outputs information representing the depth map DM generated in step S7 to an external device 4, etc. The external device 4 is, for example, a driver assistance system for the automobile 100. If the external device 4 is a driver assistance system, it can detect obstacles in front of the automobile 100, the distance to the vehicle in front, etc., and use this detection information for the automobile 100's automatic braking, cruise control, etc. Once the processing in step S8 is completed, the processing steps proceed to step S9.

[0068] In step S9, a determination is made as to whether there is a reason to stop processing. Specifically, if the control unit 31 detects an error in the imaging device 1 or a command input from the user to stop processing, it determines that there is a reason to stop processing (S9: Yes), and stops processing to terminate. On the other hand, if the control unit 31 does not detect the above-mentioned error or command input to stop processing, it determines that there is no reason to stop processing (S9: No). In this case, the processing steps return to step S1. Then, the same processing is repeated from step S1.

[0069] According to the imaging device 1 of this embodiment 1, a second exposure level is newly determined based on the brightness values ​​of multiple pixels constituting the first image obtained by first encoded imaging at a first exposure level, and the second image obtained by second encoded imaging at the second exposure level is encoded to estimate the depth of the subject. As a result, it is possible to reduce the region where the brightness value is saturated in the image to be encoded, and the accuracy of the depth estimation of the subject can be stably increased regardless of the brightness of the subject. This makes it possible to realize a more practical DFD technology.

[0070] Furthermore, according to the imaging device 1 of Embodiment 1, the second exposure level is determined based on feature quantities related to the brightness levels of multiple pixels constituting the first captured image. Therefore, it is possible to appropriately determine how much the second exposure level should be changed from the first exposure level.

[0071] Furthermore, according to the imaging device 1 of Embodiment 1, the above-mentioned feature quantity is the number of pixels among the multiple pixels constituting the first captured image whose brightness value is the maximum brightness value corresponding to the highest grayscale. Therefore, a value that directly represents the extent of the region where the brightness value is saturated in the first captured image can be used as the feature quantity, and a more appropriate second exposure level can be determined.

[0072] Furthermore, according to the imaging device 1 of Embodiment 1, the above feature quantity is defined as the number of pixels among all pixels constituting the first captured image whose brightness value is less than or equal to the average brightness value of all pixels plus three times the standard deviation of the brightness values, and whose brightness value is equal to the maximum brightness value. Therefore, pixels that have reached the maximum brightness value due to external noise, malfunction of the photoelectric conversion element, etc., can be excluded from the counting target, and a more appropriate feature quantity can be obtained.

[0073] Furthermore, according to the imaging device 1 of Embodiment 1, the second exposure level is determined by referring to a table that associates feature quantities with a second exposure level or a method for determining the second exposure level, and using the method for determining the second exposure level corresponding to the feature quantity. Therefore, the second exposure level can be determined without performing complex calculations.

[0074] Furthermore, according to the imaging device 1 of Embodiment 1, a smaller second exposure level is determined as the feature quantity increases. Therefore, it is possible to determine a second exposure level that can more accurately reduce the saturation region of the brightness value in the captured image.

[0075] Furthermore, according to the imaging device 1 of Embodiment 1, the encoding imaging system includes an aperture and an image sensor, and the first exposure level and the second exposure level are determined by a combination of the size of the aperture opening and the opening time of the electronic shutter on the image sensor. Therefore, the exposure level can be changed without providing a physical shutter, which reduces component costs and saves space.

[0076] Furthermore, according to the imaging device 1 of Embodiment 1, the encoding imaging system includes an encoding mask, and the encoding mask includes a liquid crystal panel. Therefore, the region in the liquid crystal panel that is in a light-transmitting state can be arbitrarily set, and the mask pattern can be easily switched, the presence or absence of the mask can be switched, and the range of encoding imaging conditions can be broadened.

[0077] Furthermore, according to the imaging device 1 of Embodiment 1, one or more processors perform output processing to output the estimated depth of the subject to an external device. Therefore, not only can the depth of the subject be estimated, but meaningful information can be provided to the user by having the external device analyze the depth of the subject.

[0078] Furthermore, according to the imaging device 1 of Embodiment 1, one of its external devices is a device that assists in the driving of a vehicle. Therefore, even with an imaging device 1 having a monocular optical system, it is possible to implement collision mitigation braking, adaptive cruise control, and the like based on the depth estimation of the subject.

[0079] Furthermore, according to the imaging device 1 of Embodiment 1, the encoded imaging system is installed on the vehicle, and the subject is an object in front of the vehicle. Therefore, it is possible to estimate the distance to objects such as other vehicles traveling in front of the vehicle or objects crossing in front of the vehicle, and provide meaningful information to the person riding in the vehicle.

[0080] <Example 1> A modification 1 of the above embodiment 1 will now be described. In the above embodiment, the maximum brightness pixel count Tz is used as the feature quantity F relating to the distribution of brightness values ​​of all pixels in the captured image. Furthermore, the upper brightness value Bu, which represents the upper limit of the brightness value of the pixel to be counted as the maximum brightness pixel, is set to the average value Ba of the brightness values ​​of all pixels in the captured image + standard deviation σ × 3. However, the upper brightness value Bu is not limited to this embodiment.

[0081] Figure 9 is a diagram illustrating a modified example of Embodiment 1. The upper brightness value Bu may be, for example, the mean value Ba + standard deviation σ × 2 of the brightness values ​​of all pixels in the captured image, as shown in the histogram H3 in Figure 9. Assuming that the brightness values ​​follow a normal distribution, the range defined by the mean ± standard deviation × 2 is statistically a range that includes 95% of the entire data. Therefore, even if pixels with brightness values ​​less than or equal to the upper brightness value Bu defined in this way are counted as the maximum brightness pixel count Tz, it is possible to exclude maximum brightness pixels caused by noise, etc., from the count as much as possible, and to obtain a maximum brightness pixel count Tz that sufficiently reflects the trend of the brightness value distribution. The upper brightness value Bu may also be the mean value Ba + standard deviation σ × N (2 ≤ N ≤ 3) of the brightness values ​​of all pixels in the captured image.

[0082] <Modification 2> A modification 2 of Embodiment 1 will now be described. In the above embodiment, the maximum brightness pixel count Tz is used as the feature quantity F relating to the distribution of brightness values ​​of all pixels in the captured image. However, the feature quantity F is not limited to this embodiment.

[0083] The luminance value feature F may be, for example, the average value Ba, median Bc, or mode Bg of the luminance values ​​of all pixels in the captured image, as shown in the histogram H3 in Figure 9.

[0084] Figures 10 to 12 show examples of tables relating to Modification 2 of Embodiment 1. Figure 10 shows an example of a table when the luminance value feature F is the average value Ba of the luminance value. Figure 11 shows an example of a table when the luminance value feature F is the median value Bc of the luminance value. Figure 12 also shows an example of a table when the luminance value feature F is the mode value Bg of the luminance value.

[0085] When the average value Ba of the luminance is used as the feature quantity F, for example, the second exposure level E2 can be determined by referring to the table TB1 shown in Figure 10. In this example, when the average value Ba is located near the center of the range from minimum luminance value 0 to maximum luminance value Bmax, the second exposure level E2 is determined to be the same exposure level as the first exposure level E1. Furthermore, as the average value Ba is higher than the center of the range from minimum luminance value 0 to maximum luminance value Bmax, the second exposure level E2 is determined to be lower than the first exposure level. On the other hand, as the average value Ba is lower than the center of the range from minimum luminance value 0 to maximum luminance value Bmax, the second exposure level E2 is determined to be higher than the first exposure level E1.

[0086] When the median Bc or mode Bg of the luminance value is used as the feature quantity F, the second exposure level E2 can be determined by referring to, for example, table TB2 shown in Figure 11 or table TB3 shown in Figure 12, similar to when the mean value Ba is used. That is, in these examples as well, when the median Bc or mode Bg is located near the center of the range from minimum luminance value 0 to maximum luminance value Bmax, the second exposure level E2 is determined to be the same exposure level as the first exposure level E1. Furthermore, as the median Bc or mode Bg is higher than the center of the range from minimum luminance value 0 to maximum luminance value Bmax, the second exposure level E2 is determined to be lower than the first exposure level. On the other hand, as the median Bc or mode Bg is lower than the center of the range from minimum luminance value 0 to maximum luminance value Bmax, the second exposure level E2 is determined to be higher than the first exposure level E1.

[0087] Thus, as the luminance value feature F, the mean value Ba, median value Bc, or mode value Bg of the luminance values ​​of all pixels in the captured image can be used. In this case, as a method for determining the second exposure level E2, a method can be adopted in which the higher these values ​​are, the smaller the second exposure level E2 will be compared to the first exposure level E1, and the lower these values ​​are, the larger the second exposure level E2 will be compared to the first exposure level E1. Even with such feature quantities and methods for determining the second exposure level, it is possible to reduce the pixel regions where the luminance values ​​are saturated in the captured image used for depth estimation of the subject, and to estimate the depth of the subject with high accuracy.

[0088] <Variation 3> Regarding the processing flow in imaging device 1, another example will be described. Figure 13 is a diagram showing the processing flow in the imaging device 1 according to Modification 3 of Embodiment 1. The processing flow shown in Figure 13 consists of steps S1 to S10, and each of the processes in steps S1 to S9 is the same as in the embodiment of Embodiment 1 described above. In this modification, step S10 is added between steps S3 and S4. Here, the processing of steps S1 to S9 will be omitted except for the necessary parts.

[0089] In this modified example, in step S1, encoding imaging of the subject 90 is performed at the initially set first exposure level E1, and in step S2, a histogram of the luminance values ​​in the image obtained by the encoding imaging is generated. Then, in step S3, feature quantities related to the distribution of luminance values ​​are calculated, and once this process is completed, the processing steps proceed to step S10.

[0090] In step S10, it is determined whether the exposure level is excessive. Specifically, the control unit 31 determines, based on the calculated feature quantity F, whether the first exposure level E1 is excessive relative to the brightness of the subject 90. For example, if the maximum brightness pixel count Tz is used as the feature quantity F, and this maximum brightness pixel count Tz is greater than or equal to a predetermined percentage of the total number of pixels in the captured image, for example, 0.1%, then the first exposure level E1 is determined to be excessive. If it is less than the predetermined percentage, then the first exposure level E1 is determined not to be excessive.

[0091] If, in step S10, it is determined that the first exposure level E1 is excessive (S10: Yes), the processing steps proceed to step S4. Then, the second exposure level E2 is determined (S4), encoded imaging is performed at the second exposure level E2 (S5), and the depth of the subject 90 is estimated based on the decoding of the image obtained from the encoded imaging at the second exposure level E2 (S6).

[0092] On the other hand, if it is determined in step S10 that the first exposure level E1 is not excessive (S10: No), the processing steps proceed to step S6. Then, depth estimation of the subject is performed (S6) based on the decoding of the captured image obtained by encoding imaging at the first exposure level E1.

[0093] According to this modified example 3, if the initially set first exposure level E1 is determined not to be excessive for the brightness of the subject 90, the depth of the subject is estimated based on the captured image obtained by encoding imaging at the first exposure level E1. In other words, encoding imaging at the second exposure level E2 is not performed again. This reduces the processing load on the imaging device 1.

[0094] <Modification 4> Regarding the processing flow in imaging device 1, another example will be described. Figure 14 is a diagram showing the processing flow in the imaging device 1 according to a modified example 4 of Embodiment 1. The processing flow shown in Figure 14 consists of steps S2-S3 and S6-S13, and each of the processes in steps S2-S3 and S6-S9 is the same as the process in the embodiment of Embodiment 1 described above. In this modified example, steps S11-S12 are added before step S2, and steps S10 and S13 are added between steps S3 and S6. Here, the processing in steps S2-S3 and S6-S9 will be omitted from explanation except for the necessary parts.

[0095] In this modified example, step S11 initializes the exposure level. Specifically, the exposure level determination unit 35 determines and sets the exposure level in the encoded imaging to the first exposure level E1. Once the processing in step S11 is completed, the processing steps proceed to step S12.

[0096] In step S12, encoded imaging is performed. Specifically, the encoded imaging control unit 32 controls the imaging system control unit 11 to cause the encoded imaging system 2 to perform encoded imaging at the set exposure level. Once the processing in step S12 is completed, the processing steps proceed to step S2.

[0097] In step S2, a histogram H of the brightness values ​​in the captured image is generated. Next, in step S3, a feature quantity F related to the distribution of brightness values ​​is calculated. Once the processing in step S3 is complete, the processing steps proceed to step S10.

[0098] In step S10, it is determined whether the exposure level is excessive. Specifically, the control unit 31 determines, based on the calculated feature quantity F, whether the set exposure level E is excessive for the brightness of the subject 90. For example, if the maximum brightness pixel count Tz is used as the feature quantity F, and this maximum brightness pixel count Tz is equal to or greater than a predetermined percentage of the total number of pixels in the captured image, for example, 0.1%, then the set exposure level E is determined to be excessive. If it is less than the predetermined percentage, then the set exposure level E is determined to be not excessive.

[0099] If, in step S10, it is determined that the exposure level E is excessive (S10: Yes), the processing steps proceed to step S13. Then, the second exposure level E2 is determined (S4).

[0100] On the other hand, if it is determined in step S10 that the exposure level E is not excessive (S10: No), the processing steps proceed to step S6. Then, depth estimation of the subject is performed (S6) based on the decoding of the image obtained by the above encoded imaging.

[0101] In step S13, the exposure level is determined based on the feature quantity. Specifically, the exposure level determination unit 35 determines a new exposure level E based on the feature quantity F so that pixel regions where the brightness value is saturated in the captured image are suppressed. The method for determining the feature quantity and exposure level may be the same as in the embodiment 1 or modification 2 of the above embodiment. Once the processing in step S13 is completed, the processing steps return to step S12.

[0102] According to this modified example 4, the exposure level is repeatedly adjusted until the pixel region where the brightness value is saturated in the image obtained by encoded imaging falls below a predetermined amount. As a result, the pixel region where the brightness value is saturated can be reliably reduced to an acceptable level, and the accuracy of depth estimation of the subject can be maintained at a high and stable level.

[0103] According to this embodiment and its modified form, the imaging device 1 first encodes and images the subject 90 at a first exposure level using the encoding imaging system 2, calculates feature quantities related to the distribution of brightness values ​​in the obtained image P1, and determines a second exposure level based on the calculated feature quantities. Then, the imaging device 1 encodes and images the subject at the second exposure level, decodes the obtained image, and performs depth estimation of the subject.

[0104] By controlling the exposure level in such encoded imaging with the imaging device 1, it is possible to obtain an image with fewer pixel areas where the brightness value is saturated, regardless of the brightness of the subject, thereby enabling highly accurate depth estimation of the subject and realizing a more practical DFD (Data Filtration Display) technology.

[0105] Furthermore, a method for estimating the depth of a subject according to the processing flow in the imaging device 1 is also one embodiment of the present invention. That is, one embodiment of the present invention of a method for estimating the depth of a subject involves encoding and imaging a subject at a first exposure level using an encoding imaging system, calculating feature quantities related to the distribution of brightness values ​​in the image obtained by the encoding and imaging, determining a second exposure level based on the calculated feature quantities, encoding and imaging at the determined second exposure level, decoding the image obtained by the encoding and imaging, and estimating the depth of the subject.

[0106] Furthermore, one embodiment of the present invention also includes a program for a computer or one or more processors contained in a computer to execute a process that controls an encoding imaging system so that a subject is encoded and imaged at a first exposure level, calculates feature quantities relating to the distribution of brightness values ​​in the obtained imaged image, determines a second exposure level based on the calculated feature quantities, controls the encoding imaging system so that the subject is encoded and imaged at the second exposure level, decodes the imaged image obtained by said encoding imaging, and performs depth estimation of the subject, as well as a physical computer-readable storage medium that non-temporarily stores said program.

[0107] Furthermore, a program for causing a computer or one or more processors contained in a computer to function as at least an encoding imaging control unit 32, a feature quantity calculation unit 34, an exposure level determination unit 35, and a decoding processing unit 37, and a physical computer-readable storage medium for non-temporarily storing the program are also embodiments of the present invention.

[0108] Although various embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications are included. Furthermore, the embodiments described above are described in detail for the purpose of explaining the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these are within the scope of the present invention. Furthermore, the numerical values ​​and other figures included in the text and figures are merely examples, and using different ones will not impair the effects of the present invention.

[0109] For example, in the above embodiment, table TB is used to associate feature quantities F with methods for determining exposure level E, and the exposure level determination unit 35 refers to table TB, identifies an exposure level determination method based on feature quantities F, and determines the exposure level E using the identified determination method. However, table TB may be used to directly associate feature quantities F with exposure level E, and the exposure level determination unit 35 may refer to table TB to directly determine the exposure level E from feature quantities F. Alternatively, a function may be prepared that takes feature quantities F as an input parameter and exposure level E as an output parameter, and the exposure level determination unit 35 may use this function to directly derive and determine the exposure level E from feature quantities F.

[0110] Furthermore, in the above embodiment, for example, a histogram of luminance values ​​is generated and the feature quantity F is calculated using the generated histogram. However, the feature quantity F may be calculated or identified directly from the luminance value data without generating a histogram.

[0111] Furthermore, although the above embodiment shows the imaging device 1 installed in an automobile, the imaging device 1 may also be installed in vehicles other than automobiles, such as railway or monorail trains, motorcycles, bicycles, etc. In such installation examples, the imaging device 1 will have the same effects as in the above embodiment and can be used, for example, in driver assistance technology. [Explanation of Symbols]

[0112] 1...Imaging device, 2...Encoded imaging system, 3...Arithmetic control processing unit, 4...External device, 11...Imaging system control unit, 21...Mask, 22...Optical system, 23...Image sensor, 23a...Light-receiving surface, 31...General control unit, 32...Encoded imaging control unit, 33...Brightness value histogram generation unit, 34...Feature calculation unit, 35...Exposure level determination unit, 36...Table storage unit, 37...Decoding processing unit, 38...Depth map generation unit, 90...Subject, 100...Automobile, 301...Processor, 302...Memory, 303...Storage, 304...Interface, 305...Communication bus

Claims

1. Encoded imaging system, One or more processors that control the encoding imaging system, An imaging device comprising, The encoding imaging system includes an encoding mask, The aforementioned encoding mask includes a liquid crystal panel, The one or more processors described above are: A first encoding imaging process is performed to control the encoding imaging system to encode and image the subject at a first exposure level, and to obtain a first image. An exposure determination process that determines a second exposure level based on the brightness values ​​of a plurality of pixels constituting the first captured image obtained, A second encoding imaging process is performed to control the encoding imaging system to encode and image the subject at the second exposure level determined above, thereby obtaining a second image. Based on the information obtained by decoding the second image obtained above, a depth estimation process is performed to estimate the depth of each of the multiple positions on the subject from the encoded imaging system. Imaging device.

2. In the imaging apparatus according to claim 1, The exposure determination process is a process that determines the second exposure level based on feature quantities relating to the brightness levels of the plurality of pixels. Imaging device.

3. In the imaging device according to claim 2, The aforementioned feature quantity is the number of pixels among the plurality of pixels whose brightness value corresponds to the highest brightness value corresponding to the highest grayscale. Imaging device.

4. In the imaging device according to claim 3, The aforementioned feature quantity is the number of pixels among all pixels constituting the first captured image whose brightness value is less than or equal to the average value of the brightness values ​​of all pixels plus a value between three and two times the standard deviation of the brightness values, and whose brightness value is equal to the maximum brightness value. Imaging device.

5. In the imaging device according to claim 2, The aforementioned feature is the mean, median, or mode of the brightness values ​​of the plurality of pixels. Imaging device.

6. In the imaging device according to claim 2, The exposure determination process involves referring to a table in which the feature quantities are associated with the second exposure level or the method for determining the second exposure level, and determining the second exposure level using the method corresponding to the feature quantities. Imaging device.

7. In the imaging device according to claim 2, The exposure determination process is a process that determines a smaller second exposure level as the feature quantity increases. Imaging device.

8. In the imaging apparatus according to claim 1, The encoding imaging system includes an aperture and an image sensor. The first exposure level and the second exposure level are determined by a combination of the aperture size and the opening time of the electronic shutter in the image sensor. Imaging device.

9. In the imaging apparatus according to claim 1, The one or more processors perform output processing to output the estimated depth of the subject to an external device. Imaging device.

10. In the imaging device according to claim 9, The aforementioned external device is a device that assists in the driving of a vehicle. Imaging device.

11. In the imaging apparatus according to claim 1, The aforementioned encoding imaging system is installed in a vehicle, The subject is an object in front of the vehicle. Imaging device.

Images