Imaging device, subject depth estimation method, and program

Abstract

This improves the practicality of depth estimation of subjects using encoded imaging. [Solution] In an imaging device that performs encoded imaging, the processor performs the following processes: storing the captured image in memory; decoding the captured image based on the point spread function (PSF) of the imaging system for multiple PSFs with different point light source depths, and storing multiple decoded images in memory; evaluation value calculation process applying an evaluation function to each partial image region of the multiple decoded images and storing multiple evaluation values ​​in memory; and calculation using memory based on the multiple evaluation values ​​to determine the depth estimate of the subject portion corresponding to each partial image region of the captured image. When the processor performs the decoding image generation process and the evaluation value calculation process, it first performs processing for a first group of point spread functions, then performs processing for a second group of point spread functions, and determines the final depth estimate for each subject portion based on the results of these executions.

Description

Technical Field

[0001] The present invention relates to an imaging device, a subject depth estimation method, and a program.

Background Art

[0002] In the field of coded imaging, a technique called DFD (Depth From Defocus) is known. The DFD technique is a technique for estimating the distance from the optical system of an imaging device to a subject, that is, the depth or depth of the subject, based on the degree of blurring of an edge captured in an image obtained by imaging.

[0003] The DFD technique is described in, for example, Non-Patent Document 1. In the DFD technique, coded imaging is performed in which a mask called a coded aperture is arranged in the light incident region of the optical system to image a subject. Next, the coded imaging image obtained by coded imaging is subjected to a decoding process based on a point spread function unique to the coded aperture, and the depth of the subject is estimated. The point spread function is generally called a PSF (Point Spread Function), and is also referred to as a blur function, a blur spread function, a point image distribution function, and the like.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] DFD technology is still under development, and there is much room for improvement in terms of practicality. For these reasons, there is a need for more practical DFD technology. [Means for solving the problem]

[0006] The following is a summary of some of the representative inventions disclosed in this application.

[0007] A typical embodiment of the present invention comprises an imaging system, a processor, and a memory, wherein the imaging system includes a mask for forming an encoding aperture, an optical system, and an image sensor, and the processor controls the imaging system so that encoded imaging is performed, and performs an encoded imaging process to store the image obtained by the encoded imaging in the memory, a decoding process based on the point spread function of the imaging system for a plurality of point spread functions where the depth of the corresponding point light source from the imaging system is different, and stores the obtained plurality of decoded images in the memory, and applies an evaluation function to each of the plurality of decoded images for each partial image region, and obtains a plurality The imaging device performs the following: an evaluation value calculation process that stores evaluation values ​​in the memory; a depth search process that searches for depth estimates of subjects corresponding to each partial image region in the captured image based on the plurality of evaluation values; and for each of a plurality of point spread function groups that are set based on the plurality of point spread functions and have different ranges from the minimum to the maximum depth of the corresponding point light source, the decoded image generation process, the evaluation value calculation process, and the depth search process are executed sequentially, and a process is executed to determine depth estimates of subjects corresponding to each partial image region in the captured image based on the execution results of the depth search process corresponding to each of the plurality of point spread function groups.

[0008] Furthermore, a typical embodiment of the present invention includes an encoding imaging process in which a processor controls an imaging system including a mask for forming an encoding aperture, an optical system, and an image sensor so that encoding imaging is performed, and stores the image obtained by the encoding imaging in memory; a decoding image generation process in which a decoding process based on the point spread function of the imaging system is applied to the image for a plurality of point spread functions, each with a different depth from the imaging system for a corresponding point light source, and stores the obtained plurality of decoded images in memory; and an evaluation value calculation process in which an evaluation function is applied to each of the plurality of decoded images for each partial image region, and the obtained plurality of evaluation values ​​are stored in memory. The subject depth estimation method includes: executing a depth search process to search for depth estimates of subject portions corresponding to each partial image region in the captured image based on the plurality of evaluation values; sequentially executing the decoding image generation process, the evaluation value calculation process, and the depth search process for each of a plurality of point spread function groups, which are set based on the plurality of point spread functions and have different ranges from the minimum to the maximum depth of the corresponding point light source; and executing a process to determine depth estimates of subject portions corresponding to each of the plurality of point spread function groups based on the execution results of the depth search process corresponding to each of the plurality of point spread function groups.

[0009] Furthermore, a typical embodiment of the present invention involves a processor controlling an imaging system including a mask for forming an encoding aperture, an optical system, and an image sensor so that encoded imaging is performed, and performing an encoded imaging process to store the image obtained by the encoded imaging in memory; a decoded image generation process which applies a decoding process based on the point spread function of the imaging system to the image for a plurality of point spread functions where the depth of the corresponding point light source from the imaging system is different, and stores the obtained plurality of decoded images in memory; and applying an evaluation function to each of the plurality of decoded images for each partial image region, and storing the obtained plurality of evaluation values ​​in memory. This program performs a calculation process and a depth search process that searches for depth estimates of subjects corresponding to each partial image region in the captured image based on the plurality of evaluation values. It also performs the decoding image generation process, the evaluation value calculation process, and the depth search process sequentially for each of a group of point spread functions, each of which is set based on the plurality of point spread functions and has a different range from the minimum to the maximum depth of the corresponding point light source. Finally, it performs a process to determine the depth estimates of subjects corresponding to each partial image region in the captured image based on the execution results of the depth search process corresponding to each of the group of point spread functions. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the installation of the imaging device according to Embodiment 1. [Figure 2] This figure shows an example of the configuration of the imaging device according to Embodiment 1. [Figure 3] This figure shows an example of the hardware configuration of the arithmetic control processing unit according to Embodiment 1. [Figure 4] This figure shows an example of the functional block configuration of the arithmetic control processing unit according to Embodiment 1. [Figure 5] This figure shows an example of multiple point spreading functions according to Embodiment 1. [Figure 6] This figure shows an example of a group of point spreading functions according to Embodiment 1. [Figure 7]This figure shows an overview of the processing performed by the arithmetic control processing unit according to Embodiment 1. [Figure 8] This is a flowchart showing an example of the operation flow in the imaging device according to Embodiment 1. [Figure 9] This figure shows an example of the relationship between the point spread function group according to Embodiment 1 and the estimated depth of the subject. [Figure 10] This figure shows an example of setting a group of point spreading functions according to a modified example 1 of Embodiment 1. [Figure 11] This figure shows an example of the relationship between the point spread function group and the estimated depth of the subject according to a modified example 1 of Embodiment 1. [Modes for carrying out the invention]

[0011] <Background of the inventors' research> Before describing embodiments of the present invention, we will explain the basics of DFD technology and the problems identified by the present inventors.

[0012] The degree of blurring of a subject in an image obtained by encoded imaging generally depends on the point spread function determined by the imaging system of the imaging device. The point spread function is a function that defines how a point light source located at a predetermined depth from the imaging system of the imaging device is spread out and depicted in the image.

[0013] When coded imaging is performed, an imaging image in which at least a part of the subject is blurred is obtained. When a decoding process of performing inverse convolution based on a point spread function specific to the coding aperture used for the coded imaging is performed on this imaging image, a decoded image with improved blurring of the subject part located at the same depth as the depth of the point light source assumed in the point spread function is obtained. Therefore, when the decoding process is performed on the imaging image based on each of a plurality of point spread functions having different assumed depths of the point light source, a plurality of decoded images are obtained. Then, when an evaluation function for calculating an evaluation value representing the degree of improvement of blurring is applied to each partial image region in each of these plurality of decoded images, an evaluation value obtained by quantifying the degree of improvement of blurring is obtained for each partial image region in these plurality of decoded images.

[0014] If there are evaluation values for each partial image region in the plurality of decoded images, a plurality of decoded images with improved blurring of the subject part can be obtained for each depth from the imaging system. Then, based on these plurality of decoded images, by synthesizing the partial images when the blurring is most improved for each partial image region, a blurring improvement image with improved blurring of the entire subject included in the imaging image can be obtained. Also, a depth map in which the depth information of the subject part corresponding to each partial image region in this blurring improvement image is quantified can be obtained.

[0015] On the other hand, the inventors have found that there is a problem to be described below in the process of performing a decoding process based on a plurality of point spread functions on an imaging image obtained by coded imaging, calculating an evaluation value for each partial image region in the plurality of decoded images, and obtaining an estimated depth value of each subject part.

[0016] The imaging device includes an imaging system, a processor, and a memory. The imaging system includes a mask that forms an encoding aperture, an optical system such as a lens, and an imaging device. The processor executes a process of controlling the imaging system to perform encoded imaging and storing data representing the obtained imaging image in the memory. Further, the processor executes a process of performing a decoding process on the imaging image based on a plurality of point spread functions having different depths of corresponding point light sources and storing data representing the obtained plurality of decoded images in the memory. Further, the processor executes a process of applying an evaluation function to each partial image region in each of the plurality of decoded images and storing data representing the obtained plurality of evaluation values in the memory. Furthermore, the processor executes a depth search process of searching for an estimated depth value of the subject portion corresponding to each partial image region in the imaging image based on the plurality of evaluation values.

[0017] There is a requirement to minimize the capacity of the memory mounted on the imaging device from the viewpoint of minimizing the manufacturing cost of the imaging device as much as possible. On the other hand, the processor executes, using the memory, a decoding process based on a plurality of point spread functions on the imaging image and a process of calculating an evaluation value for each partial image region in each of the obtained plurality of decoded images. Further, the processor executes a process of storing data representing the obtained plurality of decoded images and evaluation values in the memory. Therefore, if the capacity of the memory is small, the processor cannot efficiently execute various processes, and it takes a relatively long time to finally obtain an estimated depth value of the subject portion corresponding to each partial image region.

[0018] An imaging device for obtaining the depth of the subject portion corresponding to each partial image region in the imaging image may be installed in a moving body such as a vehicle. In this case, it is conceivable to sequentially obtain the depth of each partial region of each subject existing in the front or periphery of the moving body by the imaging device and use the depth information of the subject for the movement support of the moving body. Also in this case, the depth information of the subject needs to be obtained at a relatively short time interval, and the process for obtaining the depth information of the subject must be executed at high speed.

[0019] Due to the circumstances described above, one of the challenges in DFD technology is to enable faster retrieval of depth information of a subject, even when the memory capacity of the imaging device is small.

[0020] In view of the above circumstances, the present inventors have devised the present invention as a result of diligent study. Embodiments of the present invention will be described below. The embodiments described below are examples for carrying out the present invention and do not limit the technical scope of the present invention. In the following embodiments, components having the same function are denoted by the same reference numerals, and repeated descriptions thereof will be omitted unless particularly necessary.

[0021] (Embodiment 1) An imaging device according to Embodiment 1 of the present application will be described.

[0022] <Hardware configuration of the imaging device> Figure 1 shows an example of the installation of the imaging device 1 according to Embodiment 1. In the figure, the z direction is the forward direction of the automobile 100.

[0023] As shown in Figure 1, the imaging device 1 is installed on a vehicle, which is an automobile 100. The imaging device 1 is positioned to encode and capture a subject 90 located in front of the automobile 100 in the z direction.

[0024] Figure 2 shows an example of the configuration of the imaging device 1 according to Embodiment 1. As shown in Figure 2, the imaging device 1 includes an imaging system 2, a processing unit 3, and an imaging system control unit 11.

[0025] The imaging system control device 11 is electrically or communicatively connected to the imaging system 2 and the arithmetic control processing unit 3. Based on the control signal CL from the arithmetic control processing unit 3, the imaging system control device 11 controls the components of the imaging system 2 to set imaging conditions for encoded imaging and to perform encoded imaging under the set imaging conditions.

[0026] The imaging system 2 is installed as part of the imaging device 1, for example, in the front of the interior of a car 100. The imaging system 2 includes a mask 21, an optical system 22, an image sensor 23, and an aperture 24.

[0027] The mask 21 forms an encoded aperture having a geometric aperture pattern. The mask 21 functions as a filter for light that enters the optical system 22 from the subject 90 and reaches the image sensor 23. The mask 21 may be placed between the optical system 22 and the image sensor 23. The mask 21 may include, for example, a liquid crystal panel. In this case, the mask 21 can form an encoded aperture having a substantially arbitrary geometric aperture pattern by controlling the light-transmitting and light-blocking regions of the liquid crystal panel.

[0028] The optical system 22 focuses the light incident from the subject 90 onto the light-receiving surface 23a of the image sensor 23 to form an image. The optical system 22 is, for example, a lens. The lens may be a single lens or a compound lens, and may be a fixed-focal-length lens or a zoom lens.

[0029] The image sensor 23 has a light-receiving surface 23a, which is composed of a plurality of photoelectric conversion elements arranged in two dimensions. The image sensor 23 converts the intensity of the received light L into an electrical signal corresponding to the amount of light received within a certain period of time, for example, for each photoelectric conversion element on the light-receiving surface 23a, and generates image data representing the captured image based on the electrical signals of each photoelectric conversion element. The image sensor 23 outputs the generated image data of the captured image to the arithmetic control processing unit 3. Alternatively, the image sensor 23 may output the photoelectrically converted electrical signals to the arithmetic control processing unit 3, and the arithmetic control processing unit 3 may generate image data of the captured image based on those electrical signals. The image sensor 23 is also called an image sensor.

[0030] The aperture 24 forms an opening of variable size. By changing the size of this opening, the aperture 24 controls the amount of light L incident from the subject 90 and received by the light-receiving surface 23a of the image sensor 23, that is, the amount of light received by the light-receiving surface 23a when performing encoded imaging. The aperture 24 is positioned between the subject 90 and the mask 21, between the mask 21 and the optical system 22, or between the optical system 22 and the image sensor 23.

[0031] The imaging system 2 has its own point spreading function. The point spreading function is a function that determines how point light sources or point images on a subject appear blurred in the captured image. The point spreading function depends on the combination of the mask 21, the optical system 22, the image sensor 23, and the aperture 24. Furthermore, there are different point spreading functions depending on the depth of the corresponding point light source from the imaging system 2.

[0032] The arithmetic control processing unit 3 sends a control signal CL to the imaging system control unit 11, causing the imaging system control unit 11 to control the imaging system 2. Specifically, for example, the arithmetic control processing unit 3 causes the imaging system control unit 11 to set imaging conditions for encoded imaging, or to perform encoded imaging under the set imaging conditions. The imaging conditions for encoded imaging include, for example, the aperture pattern of the encoding aperture and the exposure level.

[0033] The arithmetic control processing unit 3 transmits a control signal to the imaging system control unit 11, causing the imaging system control unit 11 to repeatedly perform encoded imaging for the required period. The arithmetic control processing unit 3 acquires an image P each time encoded imaging is performed. The arithmetic control processing unit 3 performs a decoding process based on a point spread function for each acquired image P. In this process, the arithmetic control processing unit 3 performs a decoding process on the image P for each point spread function, based on a plurality of point spread functions in which the depths of the corresponding point light sources differ from each other. The plurality of point spread functions described above are the point spread functions to be used when determining the depth estimate for each part of the subject in the image. That is, the plurality of point spread functions are predetermined according to the range of depth to be determined or the resolution of depth to be determined.

[0034] The arithmetic control processing unit 3 obtains multiple decoded images in which the blur is improved according to the depth of the subject portion corresponding to each sub-image region in the captured image P, through a decoding process based on these multiple point spreading functions. The arithmetic control processing unit 3 applies an evaluation function to each sub-image region in each of the obtained multiple decoded images to calculate an evaluation value representing the degree of blur improvement, thereby obtaining multiple evaluation values.

[0035] The arithmetic control processing unit 3 obtains a blur-improved image in which the blur of the entire subject 90 included in the captured image P is improved, and depth estimates from the imaging system 2 for each sub-image region of the subject in the captured image P, based on the obtained multiple decoded images and multiple evaluation values. The arithmetic control processing unit 3 generates a depth map DM by associating the depth estimates with the blur-improved image and outputs the depth map DM to an external device 4 or the like.

[0036] Figure 3 shows an example of the hardware configuration of the arithmetic control processing unit 3 according to Embodiment 1. In this embodiment, the arithmetic control processing unit 3 is, for example, a computer, a microcontroller, etc. The arithmetic control processing unit 3 has a processor 301, memory 302, storage 303, interface 304, and communication bus 305. The processor 301, memory 302, storage 303, and interface 304 are connected to each other so as to be able to communicate with each other via the communication bus 305. At least a part of the arithmetic control processing unit 3 may be composed of semiconductor devices such as FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit).

[0037] The processor 301 may include, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an MCU (Micro Controller Unit), or a GPU (Graphics Processing Unit). The processor 301 performs various calculations and processes. The processor 301 may consist of one CPU, MCU, GPU, etc., or it may consist of two or more.

[0038] Memory 302 is, for example, a semiconductor memory device. Memory 302 temporarily stores data used for processing when the processor 301 performs various processes. Memory 302 also temporarily stores various images or numerical values ​​obtained as a result of the processor 301 performing various processes.

[0039] Storage 303 can be, for example, an SSD (Solid State Drive), HDD (Hard Disk Drive), USB memory, eMMC (Embedded Multi Media Card), EPROM (Erasable Programmable Read-Only Memory), or EEPROM (Electrically Erasable PROM). Storage 303 stores the program PR. The processor 301 reads the program PR stored in storage 303, expands it into memory 302, and executes it, thereby functioning as various functional blocks. Note that storage 303 may be omitted. In this case, the program PR is stored in memory 302.

[0040] Interface 304 receives control from the control unit 31 and performs various data transmission and reception. For example, Interface 304 receives an electrical signal or image data representing the captured image P from the image sensor 23 and stores it in memory 302. Interface 304 also reads depth information of the subject 90, which represents the depth estimate of the subject portion corresponding to each partial image area in the captured image P, from memory 302 and outputs it to the external device 4. Interface 304 also reads image data representing the blur-improved image corresponding to the captured image P from memory 302 and outputs it to the external device 4. Alternatively, Interface 304 reads a depth map DM from memory 302, which associates the subject's depth information with the blur-improved image, and outputs it to the external device 4.

[0041] <Functional block configuration of the arithmetic control unit> Figure 4 shows an example of the functional block configuration of the arithmetic control processing unit 3 according to Embodiment 1. As described above, this functional block is realized by the processor 301 executing the program PR. As shown in Figure 4, the arithmetic control processing unit 3 has as functional blocks an overall control unit 31, an encoding imaging control unit 32, a point spread function storage unit 33, a decoded image generation unit 34, an evaluation value calculation processing unit 35, a depth search processing unit 36, a depth map generation unit 37, and a data storage unit 38.

[0042] The central control unit 31 controls each part of the arithmetic control processing unit 3 so that encoded imaging of the subject 90 is performed, multiple decoded images corresponding to the captured image P are acquired, and the depth of the corresponding subject portion is searched for for each partial image region in the captured image P.

[0043] The encoding and imaging control unit 32, under control from the overall control unit 31, controls the imaging system control device 11 so that encoding and imaging are performed under set conditions. The encoding and imaging control unit 32 also stores the image P obtained by encoding and imaging in the data storage unit 38. From a hardware perspective, the processor 301 stores the image data representing the image P in the memory 302.

[0044] The point spread function storage unit 33, under control from the overall control unit 31, stores multiple point spread functions PSF1, ..., PSFM used in the decoding process. From a hardware perspective, the memory 302 stores data representing these multiple point spread functions PSF1, ..., PSFM.

[0045] Figure 5 shows an example of multiple point spreading functions according to Embodiment 1. As shown in Figure 5, the multiple point spreading functions PSF1, ..., PSFM are, for example, point spreading functions where the depths of the corresponding point light sources are 5m, 6m, 7m, 9m, 11m, 14m, 17m, 21m, 27m, 33m, 41m, 52m, 64m, 80m, and 100m, respectively. Let's assume that a decoding process based on the point spreading function PSF3, where the depth of the corresponding point light source is 7m, is applied to the captured image P, and a decoded image is obtained in which the blurring in a certain part of the subject in the captured image P is improved. In this case, as described above, the depth from the imaging system 2 to that part of the subject is estimated to be 7m, the same as the depth of the corresponding point light source in the point spreading function PSF3.

[0046] The decoded image generation unit 34 receives the captured image P obtained by encoded imaging from the image sensor 23 under control from the overall control unit 31, and performs a decoding process on the captured image P. The decoding process of the captured image P is an inverse convolution process based on the point spread function PSF determined by the aperture pattern of the mask 21, the optical system 22, the image sensor 23, the aperture 24, etc. When the captured image P is subjected to a decoding process based on a predetermined point spread function PSF, a decoded image R is obtained in which the blurring of the subject portion located at the same depth as the depth of the corresponding point light source from the imaging system 2 in that predetermined point spread function PSF is improved.

[0047] The decoded image generation unit 34 performs a decoding process on the captured image P based on multiple point spread functions PSF1, ..., PSFM, each having a different depth for the corresponding point light source, to obtain multiple decoded images R1, ..., RM. The decoded image generation unit 34 stores the obtained multiple decoded images R1, ..., RM in the data storage unit 38. From a hardware perspective, the processor 301 performs a decoding process on the captured image P to obtain multiple decoded images R1, ..., RM. Then, the processor 301 stores the image data representing the obtained decoded images R1, ..., RM in the memory 302.

[0048] The evaluation value calculation processing unit 35, under control from the overall control unit 31, applies the evaluation function f to each partial image region rij when the entire image region of the decoded image R is divided into multiple parts, and calculates an evaluation value Fij that represents the degree of blur improvement. A partial image region rij is a partial image region identified by row number i and column number j. Partial image regions rij are, for example, image regions with a vertical and horizontal pixel count of 3x3, image regions with a vertical and horizontal pixel count of 5x5, etc. An image region with a vertical and horizontal pixel count of 1x1 is a single pixel itself.

[0049] The evaluation value calculation processing unit 35 applies the evaluation function f to each of the multiple decoded images R1, ..., RM, for each sub-image region rij. This yields multiple evaluation values ​​F1ij, ..., FMij. The evaluation value calculation processing unit 35 stores the obtained multiple evaluation values ​​F1ij, ..., FMij in the data storage unit 38. From a hardware perspective, the processor 301 calculates multiple evaluation values ​​F1ij, ..., FMij for each sub-image region in the multiple decoded images R1, ..., RM. The processor 301 then stores the data representing the calculated multiple evaluation values ​​F1ij, ..., FMij in the memory 302.

[0050] The depth search processing unit 36, under control from the overall control unit 31, identifies the decoded image Rsij in which the blur is best improved for each sub-image region rij, based on the calculated evaluation values ​​F1ij, ... FMij. The depth search processing unit 36 ​​also determines that the depth of the corresponding point light source in the point spread function PSFij when the decoded image Rij is obtained is the estimated depth value dsij of the subject portion represented by that sub-image region rij. That is, for each sub-image region rij, the depth search processing unit 36 ​​obtains the evaluation value Fsij, the sub-image RsPij corresponding to the sub-image region rij in the decoded image Rsij, and the estimated depth value dsij of the subject when the blur is best improved. The depth search processing unit 36 ​​stores the evaluation value Fsij, the sub-image RsPij, and the estimated depth value dsij obtained for each sub-image region rij in the data storage unit 38.

[0051] From a hardware perspective, the processor 301 identifies the evaluation value Fsij and the partial image RsPij for each partial image region rij when the blur is best improved, based on multiple evaluation values ​​F1ij, ..., FMij, and determines the depth estimate dsij of the corresponding subject area. The processor 301 then stores the data representing the evaluation value Fsij, the partial image RsPij, and the depth estimate dsij obtained for each partial image region rij in the memory 302.

[0052] The depth map generation unit 37, under control from the overall control unit 31, synthesizes the sub-images RsPij corresponding to the sub-image region rij in the decoded image Rsij with the best blur reduction for each sub-image region rij, to generate a blur-improved image Ps. The depth map generation unit 37 also generates a depth map DM by mapping the depth estimate dij for each sub-image region rij of the subject 90 included in the generated blur-improved image Ps to that sub-image region. The depth map generation unit 37 stores the generated depth map DM in the data storage unit 38.

[0053] From a hardware perspective, the processor 301 generates a depth map DM and stores the data representing the depth map DM in the memory 302. Then, the depth map generation unit 37 outputs the generated depth map DM to an external device 4 or the like, in accordance with the control from the control unit 31.

[0054] The data storage unit 38 stores various types of data. For example, the data storage unit 38 stores data representing the captured image P, data representing the decoded images R1 to RM, data representing the evaluation values ​​F1ij to FMij, data representing the depth estimate value dij, data representing the blur-improved image Ps, and data representing the depth map DM. The data storage unit 38 also stores data that should be stored during calculations or processing by each unit.

[0055] When searching for the estimated depth value dij of the subject portion corresponding to each sub-image region rij in the captured image P, it is common practice to treat the multiple point spread functions PSF1, ... PSFM that correspond to the depth range being searched as a group of point spread functions. The central control unit 31 then controls each part so that the processing of the group of point spread functions PSF1, ... PSFM is performed simultaneously when each of the above processes is carried out.

[0056] Specifically, the above processes involve: applying a decoding process to the captured image P to generate multiple decoded images R1, ..., RM; applying an evaluation function f to each sub-image region rij in each decoded image R1, ..., RM to calculate multiple evaluation values ​​F1ij, ..., FMij; and searching for the depth dij of the subject portion corresponding to the sub-image region rij based on the multiple evaluation values ​​F1ij, ..., FMij.

[0057] The reason why the processing of a group of point spread functions, PSF1, ..., PSFM, is performed simultaneously during each of the above processes is, for example, as follows: In order to search for depth estimates of each subject part, it is necessary that all evaluation values ​​F1ij, ..., FMij for each sub-image region rij are available in each of the multiple decoded images R1, ..., RM corresponding to all the point spread functions PSF1, ..., PSFM to be handled. Considering this point, it is simple and very natural to construct an algorithm in which the processing of a group of point spread functions G is performed simultaneously during each of the above processes.

[0058] On the other hand, in Embodiment 1, the control unit 31 controls the decoded image generation unit 34, the evaluation value calculation unit 35, and the depth search unit 36 ​​so that each of the above processes is performed in multiple stages. That is, the depth range to be searched in each subject portion corresponding to each partial image region rij in the captured image P is divided into multiple ranges. A specific example is described below.

[0059] For example, the processes of decoding the captured image P, applying the evaluation function f to the decoded image, and searching for depth are each executed in two separate steps. Here, we consider the case where there are a total of M point spreading functions PSF applied to the decoding process, and approximately half of the M point spreading functions PSF are N (if M is even, N is M / 2 or (M / 2)+1; if M is odd, N is (M+1) / 2). In this case, the N point spreading functions from PSF1, which has the smallest depth of the corresponding point light source, to PSFN, which has an intermediate depth, are defined as the first group of point spreading functions G1. Furthermore, the (M-N+1) point spreading functions, from the point spreading function PSFN, which represents an intermediate depth for the point light source, to the point spreading function PSFM, which represents the largest depth for the point light source, are defined as the second group of point spreading functions G2.

[0060] Figure 6 shows an example of a group of point spreading functions according to Embodiment 1. For example, let's assume that all the point spreading functions PSF1, ..., PSFM to be dealt with are point spreading functions from PSF1 to PSF15 as shown in Figure 6. That is, there are M functions = 15 functions and N functions = 8 functions.

[0061] In this case, as shown in Figure 6, the first group of point spreading functions G1 has a first depth range, which is the range from the minimum to the maximum depth of the corresponding point light source, that is, on the relatively short side, i.e., the near-field side. That is, the first group of point spreading functions G1 includes eight point spreading functions PSF1, ..., PSF8, for which the depth of the corresponding point light source is 5m, ..., 21m. The second group of point spreading functions G2 includes eight point spreading functions PSF8, ..., PSF15, for which the second depth range, which is the range from the minimum to the maximum depth of the corresponding point light source, is on the relatively long side, i.e., the far-field side.

[0062] In this example, the first and second depth ranges correspond to the ranges obtained by dividing the range from the minimum to the maximum depth of the corresponding point light sources in multiple point spread functions PSF1, ..., PSF15 into two parts, and then dividing that range into two parts. Furthermore, the first and second depth ranges are continuous.

[0063] Next, we will explain the process for obtaining the blur-reduced sub-image RsPij and the depth estimate dsij for each sub-image region rij of the captured image P. Here, we assume that the point spreading functions PSF to be handled are PSF1, ..., PSF15, as shown in Figure 6. We also assume that the first group of point spreading functions G1 includes the point spreading functions PSF1, ..., PSF8, and the second group of point spreading functions G2 includes the point spreading functions PSF8, ..., PSF15.

[0064] Figure 7 is a diagram showing an overview of the processing performed by the arithmetic control processing unit according to Embodiment 1. First, the central control unit 31 controls each part so that processing is performed for the first point spreading function group G1. Next, the central control unit 31 controls each part so that processing is performed for the second point spreading function group G2.

[0065] The decoded image generation unit 34 performs a decoding process on the captured image P based on the point spreading functions PSF1, ..., PSF8 included in the first point spreading function group G1, and generates eight decoded images R1, ..., R8. The decoded image generation unit 34 stores the generated decoded images R1, ... R8 in the data storage unit 38. That is, the processor 301 calculates the decoded images R1, ... R8 using calculations with the memory 302. Then, the processor 301 stores the data representing these decoded images in the memory 302.

[0066] Next, the evaluation value calculation processing unit 35 applies the evaluation function f to each of the decoded images R1, ..., R8, for each sub-image region rij, and calculates evaluation values ​​F1ij, ..., F8ij. The evaluation value calculation processing unit 35 stores the calculated evaluation values ​​F1ij, ..., F8ij in the data storage unit 38. That is, the processor 301 obtains the evaluation values ​​F1ij, ..., F8ij through calculations using the memory 302. Then, the processor 301 stores the data representing these evaluation values ​​in the memory 302.

[0067] Next, the depth search processing unit 36 ​​identifies the evaluation value Fsij_1, the decoded image Rsij_1, and the depth dsij_1 for each partial image region rij when the blur is improved, based on the evaluation values ​​F1ij, ..., F8ij. Then, the depth search processing unit 36 ​​stores the identified evaluation value Fsij_1, partial image RsPij_1, and depth estimate dsij_1 for each partial image region rij in the data storage unit 38. That is, the processor 301 calculates the evaluation value Fsij_1, partial image RsPij_1, and depth estimate dsij_1 for each partial image region rij using calculations in the memory 302. Then, the processor 301 stores the data representing these evaluation values, partial images, and depth estimates in the memory 302.

[0068] The control unit 31 deletes unnecessary data from the data stored in the data storage unit 38 as a result of each process related to the first point spreading function group G1. More specifically, the control unit 31 deletes from the data storage unit 38 all other data except for the evaluation value Fsij_1, the partial image RsPij_1, and the depth estimate value dsij_1 for each partial image region rij, as a result of each process related to the first point spreading function group G1.

[0069] In other words, the processor 301 frees up the memory 302, which stores data (information) obtained from each process related to the first point spread function group G1, except for the area where some data, including data used to determine the depth estimate, is stored. This other area is where data that is no longer needed is stored. As a result, the usable capacity of memory 302 increases significantly compared to before the area where the unnecessary data was stored was freed.

[0070] When processing for the first point spreading function group G1 is completed, the control unit 31 controls each part so that processing for the second point spreading function group G2 is carried out. The decoded image generation unit 34 performs a decoding process on the captured image P based on the point spreading functions PSF8,...,PSF15 included in the second point spreading function group G2, and generates eight decoded images R8,...,R15. The decoded image generation unit 34 stores the generated decoded images R8,...R15 in the data storage unit 38. That is, the processor 301 obtains the decoded images R8,...R15 by calculation using the memory 302. Then, the processor 301 stores the data representing these decoded images in the memory 302.

[0071] Next, the evaluation value calculation processing unit 35 applies the evaluation function f to each of the decoded images R8, ..., R15, for each sub-image region rij, and calculates evaluation values ​​F8ij, ..., F15ij. The evaluation value calculation processing unit 35 stores the calculated evaluation values ​​F8ij, ..., F15ij in the data storage unit 38. That is, the processor 301 obtains the evaluation values ​​F8ij, ..., F15ij through calculations using the memory 302. Then, the processor 301 stores the data representing these evaluation values ​​in the memory 302.

[0072] Next, the depth search processing unit 36 ​​identifies the evaluation value Fsij_2, partial image RsPij_2, and depth estimate dsij_2 for each partial image region rij when the blur is improved, based on the evaluation values ​​F8ij, ..., F15ij. Then, the depth search processing unit 36 ​​stores the identified evaluation value Fsij_2, partial image RsPij_2, and depth estimate dsij_2 for each partial image region rij in the data storage unit 38. That is, the processor 301 calculates the evaluation value Fsij_2, partial image RsPij_2, and depth estimate dsij_2 for each partial image region rij using calculations with the memory 302. Then, the processor 301 stores the data representing these evaluation values, partial images, and depth estimates in the memory 302.

[0073] The control unit 31 deletes unnecessary data from the data stored in the data storage unit 38 through each process related to the second point spread function group G2. More specifically, the control unit 31 deletes from the data storage unit 38 all other data except for the evaluation value Fsij_2, the partial image RsPij_2, and the depth estimate value dsij_2 for each partial image region rij, through each process related to the second point spread function group G2.

[0074] In other words, the processor 301 frees up the memory 302, which stores data (information) obtained from each process related to the second point spread function group G2, except for the area where some data, including data used to determine the depth estimate, is stored. This other area is where data that is no longer needed is stored. As a result, the usable capacity of memory 302 increases significantly compared to before the area where the unnecessary data was stored was freed.

[0075] Once processing for the second point spread function group G2 is complete, the depth search processing unit 36 ​​compares, for each partial image region rij, the evaluation value Fsij_1 obtained from processing for the first point spread function group G1 with the evaluation value Fsij_2 obtained from processing for the second point spread function group G2. Then, the depth search processing unit 36 ​​identifies the evaluation value that shows the greater improvement in image blur between evaluation value Fsij_1 and evaluation value Fsij_2 as the evaluation value Fsij_0, which represents the final evaluation value when the image blur is best improved.

[0076] The depth search processing unit 36 ​​stores in the data storage unit 38 the partial image RsPij_0 that corresponds to the identified evaluation value Fsij_0 from among the partial images RsPij_1 and RsPij_2, as the partial image RsPij_0 when the image blur is ultimately best improved. The depth search processing unit 36 ​​also stores in the data storage unit 38 the depth estimate dsij_0 that corresponds to the identified evaluation value Fsij_0 from among the depth estimates dsij_1 and dsij_2, as the depth estimate dsij_0 when the image blur is ultimately best improved. In other words, the processor 301 identifies the partial image RsPij_0 and the depth estimate dsij_0 for each partial image region rij through calculations using the memory 302. The processor 301 then stores the data representing these partial images and depth estimates in the memory 302.

[0077] The depth map generation unit 37 generates a depth map DM corresponding to the captured image P based on the partial image RsPij_0 and depth estimate dsij_0 obtained for each partial image region rij. The depth map generation unit 37 then stores the generated depth map DM in the data storage unit 38. That is, the processor 301 calculates the depth map DM using calculations with the memory 302. The processor 301 then stores the data representing this depth map in the memory 302.

[0078] Furthermore, the control unit 31 outputs the depth map DM to the external device 4. That is, the processor 301 controls the interface 304 so that the data representing the depth map DM stored in the memory 302 is output to the external device 4.

[0079] <Processing flow in imaging device> The following describes the operation flow of the imaging device 1. Figure 8 is a flowchart showing an example of the operation flow in the imaging device 1 according to Embodiment 1.

[0080] As shown in Figure 8, in step S1, encoded imaging is performed. Specifically, the encoded imaging control unit 32 sends a control signal to the imaging system control unit 11, and the imaging system control unit 11 controls the imaging system 2 based on the received control signal. The imaging system 2 performs encoded imaging of the subject 90 under the control of the imaging system control unit 11. The data representing the image P obtained by this encoded imaging is stored in the data storage unit 38.

[0081] In step S2, the captured image is decoded using a first set of point spreading functions. Specifically, the decoded image generation unit 34 applies a decoded operation to the captured image P based on a plurality of point spreading functions PSF1, ..., PSFN included in the first set of point spreading functions G1, and generates a plurality of decoded images R1, ..., RN. The decoded image generation unit 34 stores the generated plurality of decoded images R1, ..., RN in the data storage unit 38.

[0082] In step S3, a process is performed to calculate an evaluation value for each sub-image region in each decoded image. Specifically, the evaluation value calculation processing unit 35 applies an evaluation function f to each sub-image region rij of the multiple decoded images R1, ..., RN generated in step S2, and calculates evaluation values ​​F1ij, ..., FNij, which are evaluation values ​​for each decoded image and for each sub-image region. The evaluation value calculation processing unit 35 stores the data representing the calculated evaluation values ​​F1ij, ..., FNij in the data storage unit 38.

[0083] In step S4, a process is performed to search for depth for each partial image region. Specifically, the depth search processing unit 36 ​​identifies the evaluation value Fsij_1, the decoded image Rsij_1, and the depth dij_1 for each partial image region rij in the captured image P, based on the evaluation values ​​F1ij, ..., FNij calculated in step S3, when the blur is improved. Then, the depth search processing unit 36 ​​stores the identified evaluation value Fsij_1, the partial image RsPij_1, and the depth estimate dij_1 for each partial image region rij in the data storage unit 38.

[0084] In step S5, a process is performed to delete unnecessary data. Specifically, the control unit 31 deletes from the data storage unit 38 all other data from the data storage unit 38 that was stored in the data storage unit 38 by each process related to the first point spread function group G1 in steps S2 to S4, excluding the evaluation value Fsij_1, the partial image RsPij_1, and the depth estimate value dij_1 for each partial image region rij.

[0085] In step S6, the captured image is decoded using a second set of point spreading functions. Specifically, the decoded image generation unit 34 applies a decoded operation to the captured image P based on a plurality of point spreading functions PSFN, ..., PSFM included in the second set of point spreading functions G2, and generates a plurality of decoded images RN, ..., RM. The decoded image generation unit 34 stores the generated plurality of decoded images RN, ..., RM in the data storage unit 38.

[0086] In step S7, a process is performed to calculate an evaluation value for each sub-image region in each decoded image. Specifically, the evaluation value calculation processing unit 35 applies the evaluation function f to each sub-image region rij for each of the multiple decoded images RN, ..., RM obtained in step S6, and calculates evaluation values ​​FNij, ..., FMij, which are evaluation values ​​for each decoded image and for each sub-image region. The evaluation value calculation processing unit 35 stores the data representing the calculated evaluation values ​​FNij, ..., FMij in the data storage unit 38.

[0087] In step S8, a depth search process is performed for each partial image region. Specifically, the depth search processing unit 36 ​​identifies the evaluation value Fsij_2, the decoded image Rsij_2, and the depth dij_2 for each partial image region rij in the captured image P, based on the evaluation values ​​FNij, ..., FMij obtained in step S7, when the blur is improved. Then, the depth search processing unit 36 ​​stores the identified evaluation value Fsij_2, the partial image RsPij_2, and the depth estimate dij_2 for each partial image region rij in the data storage unit 38.

[0088] In step S9, a process is performed to delete unnecessary data. Specifically, the control unit 31 deletes from the data storage unit 38 all other data except for the evaluation value Fsij_2, the partial image RsPij_2, and the depth estimate value dij_2 data for each partial image region rij, which were stored in the data storage unit 38 by the processing related to the second point spread function group G2 in steps S6 to S8.

[0089] In step S10, a process is performed to identify the evaluation value when the blur of the image is best improved for each partial image region. Specifically, the depth search processing unit 36 ​​compares the evaluation value Fsij_1 obtained from processing the first point spread function group G1 with the evaluation value Fsij_2 obtained from processing the second point spread function group G2 for each partial image region rij. Then, the depth search processing unit 36 ​​identifies the evaluation value that shows the greater improvement in image blur between evaluation value Fsij_1 and evaluation value Fsij_2 as the evaluation value Fsij_0 when the blur of the image is ultimately best improved.

[0090] In step S11, a process is performed to determine the blur-improved image of the captured image and the final depth estimate for each partial image region. Specifically, the depth search processing unit 36 ​​stores in the data storage unit 38 the partial image RsPij_0, which corresponds to the identified evaluation value Fsij_0, from among partial images RsPij_1 and RsPij_2, as the partial image RsPij_0 when the blur of the image is finally improved the most. In addition, the depth search processing unit 36 ​​stores in the data storage unit 38 the depth estimate dij_0, which corresponds to the identified evaluation value Fij_0, from among depth estimate dij_1 and depth estimate dij_2, as the depth estimate dij_0 when the blur of the image is finally improved the most.

[0091] In step S12, a process is performed to generate and output a depth map. Specifically, the depth map generation unit 37 generates a depth map DM corresponding to the captured image P based on the partial image RsPij_0 and depth estimate dij_0 obtained for each partial image region rij. The depth map generation unit 37 then stores the generated depth map DM in the data storage unit 38. The control unit 31 outputs the depth map DM stored in the data storage unit 38 to the external device 4.

[0092] In step S13, a process is performed to determine whether or not there is a reason to stop the process. Specifically, the control unit 31 determines, based on various signals, whether or not there is a reason to stop the process of causing the imaging system to perform encoded imaging and generate a depth map. The various signals include, for example, an instruction signal to terminate the process and an error signal in the imaging device 1. If it is determined that there is a reason to stop the process (step S13: Yes), the process is terminated. On the other hand, if it is determined that there is no reason to stop the process (step S14: No), the process steps return to step S1.

[0093] According to the imaging device 1 of this embodiment 1, the processor 301 sequentially executes a process to search for the depth estimate of the subject portion for each partial image region rij in the captured image P, divided into sets of point spread functions. Then, the processor 301 executes a process to obtain the final depth information of the subject based on the information obtained by the sequential execution. Therefore, even if the number of point spread functions to be handled is large relative to the capacity of the memory 302, memory capacity constraints can be suppressed, and the memory 302 can be used efficiently. As a result, in DFD technology, even if the capacity of the memory 302 installed in the imaging device 1 is small, the depth information of the subject can be obtained at a faster speed. This makes it possible to realize a more practical DFD technology.

[0094] Furthermore, according to the imaging device 1 of Embodiment 1, the imaging system includes a mask 21 that forms an encoding aperture, and the mask 21 includes a liquid crystal panel. Therefore, by arbitrarily setting the regions in the liquid crystal panel that are in a light-transmitting state and regions that are in a light-blocking state, it is possible to easily switch the aperture pattern of the mask 21, switch the presence or absence of the mask 21, and broaden the range of encoding imaging conditions.

[0095] Furthermore, according to the imaging device 1 of Embodiment 1, the processor 301 performs processing to output the estimated depth information of the subject to the external device 4. Therefore, not only is the depth of the subject estimated, but the depth of the subject is also analyzed by the external device 4, providing meaningful information to the user.

[0096] Furthermore, according to the imaging device 1 of Embodiment 1, one of its external devices 4 is a device that assists in the driving of a vehicle. Therefore, even with an imaging device 1 having a monocular optical system, it is possible to implement collision mitigation braking, adaptive cruise control, and the like based on the estimated depth value dij of the subject.

[0097] Furthermore, according to the imaging device 1 of Embodiment 1, the imaging system 2 is installed on the vehicle, and the subject is an object in front of the vehicle. Therefore, it is possible to estimate the distance to objects such as other vehicles traveling in front of the vehicle or objects crossing in front of the vehicle, and provide meaningful information to the person riding in the vehicle.

[0098] <Example 1> A modified example of Embodiment 1 will now be described.

[0099] Generally, the estimated depth of a subject contains errors from the actual depth. Therefore, when setting up multiple point spread functions, if the depth ranges of the corresponding point light sources in the point spread functions do not overlap, a depth range will be created where the depth of the subject cannot be estimated accurately. The reason for this is explained below.

[0100] For example, as shown in FIG. 6, assume that the depths of the corresponding point light sources in the plurality of point spread functions PSF to be handled are 5 m, 6 m, 7 m, 9 m, 11 m, 14 m, 17 m, 21 m, 27 m, 33 m, 41 m, 52 m, 64 m, 80 m, and 100 m, respectively. Also, when obtaining the depth estimation value for a subject whose actual depth from the imaging system 2 is in the vicinity of 21 m, assume that the depth estimation value of the subject varies within the range of the actual depth ±1.0 m. In this case, the depth estimation value of the subject will be obtained within the range of 20 m to 22 m.

[0101] Here, as shown in FIG. 6, the point spread function PSF with the depth of the corresponding point light source ≤21 m is defined as the first point spread function group G1, and the point spread function PSF with the depth of the corresponding point light source ≥21 m is defined as the second point spread function group G2. That is, the first point spread function group G1 is composed of point spread functions with the depths of the corresponding point light sources being 5 m, …, 21 m. Also, the second point spread function group G2 is composed of point spread functions with the depths of the corresponding point light sources being 21 m, …, 100 m.

[0102] FIG. 9 is a diagram showing an example of the relationship between the point spread function group according to Embodiment 1 and the estimated depth of the subject.

[0103] Generally, when obtaining the depth estimation value of a subject, within the range of the depth of the corresponding point light source in the point spread function PSF, interpolation processing may be performed, but basically, extrapolation processing is not performed.

[0104] Then, as shown in FIG. 9, since the depth corresponding to the PSF8 on the farthest distance side among the first point spread function group G1 is 21 m, the depth estimation value obtained within an error of ±1 m by the first point spread function group G1 will be 20 m or less. That is, in the depth estimation process based on the first point spread function group G1, for a subject whose actual depth D satisfies 20 m < D ≤ 21 m, it is difficult to accurately obtain the depth estimation value. For a subject whose actual depth D satisfies 20 m < D ≤ 21 m, it is not covered by the second point spread function group G2 either.

[0105] Also, as shown in FIG. 9, since the depth corresponding to the PSF8 closest to the near side in the second group of point spread functions G2 is 21 m, the depth estimate obtained within an error of ±1 m by the second group of point spread functions G2 is 22 m or more. That is, in the depth estimation process based on the second group of point spread functions G2, it is difficult to accurately obtain the depth estimate for a subject whose actual depth D satisfies 21 m ≤ D < 22 m. For a subject whose actual depth D satisfies 21 m ≤ D < 22 m, it is not covered by the first group of point spread functions G1 either.

[0106] As a result, for a subject whose actual depth D is in the range of 20 m < D < 22 m, it cannot be covered by the first group of point spread functions G1 and the second group of point spread functions G2, and it is difficult to accurately estimate the depth.

[0107] Therefore, in the first modification of Embodiment 1, the first depth range, which is the range of the depth of the corresponding point light source in the point spread function included in the first group of point spread functions G1, and the second depth range, which is the range of the depth of the corresponding point light source in the point spread function included in the second group of point spread functions G2, are set to partially overlap.

[0108] For example, consider a case where there are a total of M point spread functions PSF to be applied to the decoding process, and the number of N point spread functions, which is approximately half of the total number of point spread functions PSF (when M is even, N is M / 2 or (M / 2)+1; when M is odd, N is (M+1) / 2). In this case, N + 1 point spread functions from the point spread function PSF1 with the smallest depth of the corresponding point light source to the point spread function PSF(N+1) with an intermediate depth of the corresponding point light source are set as the first group of point spread functions G1.

[0109] Furthermore, the (M-N+2) point spreading functions from the point spreading function PSF(N-1), which represents an intermediate depth for the point light source, to the point spreading function PSFM, which represents the largest depth for the point light source, are defined as the second group of point spreading functions G2. Alternatively, the (M-N+1) point spreading functions from the point spreading function PSFN, which represents an intermediate depth for the point light source, to the point spreading function PSFM, which represents the largest depth for the point light source, are defined as the second group of point spreading functions G2.

[0110] Figure 10 shows an example of setting a group of point spreading functions according to Modification 1 of Embodiment 1. For example, as shown in Figure 10, let's assume that the depth of the corresponding point light source in the point spreading function is 5m, 6m, ..., 100m. In this case, for example, the first group of point spreading functions G1 is set to nine point spreading functions for which the depth of the corresponding point light source ranges from 5m to 27m. The second group of point spreading functions G2 is set to nine point spreading functions for which the depth of the corresponding point light source ranges from 17m to 100m. That is, the range of change of the depth of the corresponding point light source in the point spreading functions included in the first group of point spreading functions G1 and the range of change of the depth of the corresponding point light source in the point spreading functions included in the second group of point spreading functions G2 overlap in the range from 17m to 27m.

[0111] Figure 11 shows an example of the relationship between the point spread function group and the estimated depth of the subject according to Modification 1 of Embodiment 1. When the first point spread function group G1 and the second point spread function group G2 are set as shown in Figure 10, the relationship between the point spread function group and the estimated depth of the subject is as shown in Figure 11. Here, when calculating the depth estimate for a subject whose actual depth from the imaging system 2 is around 27m, it is assumed that the depth estimate of the subject will vary within the range of the actual depth ± ΔL1. Also, when calculating the depth estimate for a subject whose actual depth from the imaging system 2 is around 17m, it is assumed that the depth estimate of the subject will vary within the range of the actual depth ± ΔL2.

[0112] As shown in FIG. 11, the depth D of the subject that is difficult to accurately estimate based on the first point spread function group G1 is the depth D satisfying 27m - ΔL1

[0113] However, when there is a partial overlap in the depth range of the corresponding point light sources in the point spread function between the first point spread function group G1 and the second point spread function group G2, the processing amount will increase by that amount compared to the non - overlapping case. Therefore, when prioritizing the processing speed over the high accuracy in the process of estimating the depth of the subject, it is advisable not to overlap the depth ranges of the corresponding point light sources in the point spread function between the first point spread function group G1 and the second point spread function group G2 (Embodiment 1). On the other hand, when prioritizing the high accuracy over the processing speed in the process of estimating the depth of the subject, it is advisable to overlap the depth ranges of the corresponding point light sources in the point spread function between the first point spread function group G1 and the second point spread function group G2 (Modified Example 1 of Embodiment 1).

[0114] <Modified Example 2> A modification 2 of Embodiment 1 will now be described. In Modification 2, the decoding process applied to the captured image P, the process of applying the evaluation function f to the decoded image, and the process of searching for depth are each executed in three or more separate steps. That is, three or more point spread function groups are set. In particular, when the number of point spread functions PSF to be handled is considerably large relative to the capacity of memory 302, dividing these processes into three or more steps makes the calculations using memory 302 more efficient, and makes it possible to obtain depth estimates for each partial image region in the captured image in a shorter time.

[0115] In the modified example 2, the depth ranges of the corresponding point light sources in the point spreading functions may or may not overlap between adjacent groups of point spreading functions.

[0116] (Other embodiments) A method for estimating the depth of a subject according to the processing flow in the imaging device 1 is also one embodiment of the present invention. That is, in one embodiment of the present invention, the method for estimating the depth of a subject involves a processor controlling an imaging system including a mask that forms an encoding aperture, an optical system, and an image sensor so that encoded imaging is performed, an encoded imaging process in which the image obtained by encoded imaging is stored in memory, a decoding image generation process in which a decoding process based on the point spread function of the imaging system is applied to the image for multiple point spread functions, each with a different depth from the imaging system of the corresponding point light source, and the obtained multiple decoded images are stored in memory, and an evaluation function is applied to each of the multiple decoded images for each partial image region, and the obtained multiple evaluation values ​​are stored in memory. This is a subject depth estimation method that includes performing an evaluation value calculation process, a depth search process that searches for depth estimates of the subject portion corresponding to each partial image region in the captured image based on multiple evaluation values, sequentially performing a decoded image generation process, an evaluation value calculation process, and a depth search process for each of a group of point spread functions that are set based on multiple point spread functions and have different ranges from the minimum to the maximum depth of the corresponding point light source, and determining the depth estimates of the subject portion corresponding to each partial image region in the captured image based on the execution results of the depth search process corresponding to each of the group of point spread functions.

[0117] Furthermore, the system controls an imaging system including a mask forming an encoding aperture, an optical system, and an image sensor so that encoded imaging is performed on a computer, microcontroller, or a processor contained therein; the system performs an encoded imaging process to store the image obtained by encoded imaging in memory; the system performs a decoding process based on the point spread function of the imaging system for multiple point spread functions, each with a different depth from the imaging system for the corresponding point light source; the system performs a decoding image generation process to store the obtained multiple decoded images in memory; the system performs an evaluation function for each partial image region in each of the multiple decoded images; the system performs an evaluation value calculation process to store the obtained multiple evaluation values ​​in memory; and the system performs a decoding image generation process based on the multiple evaluation values. A program and a physical computer-readable storage medium that non-temporarily stores the program are also one embodiment of the present invention. The program performs a depth search process to search for depth estimates of subjects corresponding to each partial image region in an captured image, and sequentially performs a decoding image generation process, an evaluation value calculation process, and a depth search process for each of a group of point spread functions, each of which is set based on a group of point spread functions and has a different range from the minimum to the maximum depth of the corresponding point light source, and determines the depth estimate of subjects corresponding to each partial image region in an captured image based on the results of the depth search process corresponding to each of the group of point spread functions.

[0118] Furthermore, a program for causing a computer including a processor and memory, a microcontroller, etc. to function as at least an encoding imaging control unit 32, a point spreading function storage unit 33, a decoding image generation unit 34, an evaluation value calculation processing unit 35, a depth search processing unit 36, and a data storage unit 38, and a physical computer-readable storage medium for non-temporarily storing the program are also embodiments of the present invention.

[0119] Although various embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications are included. Furthermore, the embodiments described above are described in detail for the purpose of explaining the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these are within the scope of the present invention. Furthermore, the numerical values ​​and other figures included in the text and figures are merely examples, and using different ones will not impair the effects of the present invention.

[0120] For example, although the above embodiment shows the imaging device 1 installed in an automobile, the imaging device 1 may also be installed in a moving object other than an automobile. Examples of moving objects include railway or monorail trains, motorcycles, bicycles, etc. Even in examples of installation on such moving objects, the imaging device 1 will have the same effects as in the above embodiment and can be used, for example, in driving assistance technology for moving objects. [Explanation of symbols]

[0121] 1...Imaging device, 2...Imaging system, 3...Arithmetic control processing unit, 4...External device, 11...Imaging system control unit, 21...Mask, 22...Optical system, 23...Image sensor, 23a...Light receiving surface, 31...General control unit, 32...Encoded imaging control unit, 33...Point spread function memory unit, 34...Decoded image generation unit, 35...Evaluation value calculation processing unit, 36...Depth search processing unit, 37...Depth map generation unit, 38...Data storage unit, 90...Subject, 100...Automobile, 301...Processor, 302...Memory, 303...Storage, 304...Interface, 305...Communication bus

Claims

1. Imaging system, Processor and Equipped with memory, The imaging system includes a mask that forms an encoding aperture, an optical system, and an image sensor. The aforementioned processor, Encoded imaging process: Controls the imaging system so that encoded imaging is performed, and stores the image obtained by the encoded imaging in the memory. A decoding image generation process is performed on the captured image for multiple point spreading functions, each with a different depth from the imaging system for the corresponding point light source, based on the point spreading function of the imaging system, and the multiple decoded images obtained are stored in the memory. An evaluation value calculation process which applies an evaluation function to each of the multiple decoded images and stores the obtained evaluation values ​​in the memory, Based on the aforementioned multiple evaluation values, a depth search process is performed to search for an estimated depth value of the subject corresponding to each partial image region in the captured image. Based on the aforementioned plurality of point spreading functions, for each of the plurality of point spreading function groups, each having a different range from the minimum to the maximum depth of the corresponding point light source, the decoded image generation process, the evaluation value calculation process, and the depth search process are executed sequentially. Based on the results of the depth search process corresponding to each of the aforementioned group of point spread functions, a process is executed to determine the estimated depth of the subject portion corresponding to each partial image region in the captured image. Imaging device.

2. In the imaging apparatus according to claim 1, The processor executes a process to free up the memory region where information obtained by sequentially executing the decoding image generation process, the evaluation value calculation process, and the depth search process for at least one of the point spreading function groups is stored, excluding the region where some information, including information used to determine the depth estimate, is stored. Imaging device.

3. In the imaging device according to claim 2, The aforementioned group of point spreading functions includes a first group of point spreading functions and a second group of point spreading functions. In the two or more point spreading functions that constitute the first group of point spreading functions, the first depth range, which is the range from the minimum to the maximum depth of the corresponding point light source, is on the relatively shorter side. In the two or more point spreading functions that constitute the second group of point spreading functions, the second depth range, which is the range from the minimum to the maximum depth of the corresponding point light source, is on the relatively longer side. Imaging device.

4. In the imaging device according to claim 3, The first depth range and the second depth range are continuous. Imaging device.

5. In the imaging device according to claim 3, The first depth range and the second depth range partially overlap. Imaging device.

6. In the imaging device according to claim 3, The first depth range and the second depth range are the ranges corresponding to one division and the other division when the range from the minimum to the maximum depth of the corresponding point light source in the plurality of point spreading functions is divided into two parts. Imaging device.

7. In the imaging apparatus according to claim 1, The mask includes a liquid crystal panel that forms the encoding aperture. Imaging device.

8. In the imaging apparatus according to claim 1, The aforementioned evaluation function is a function that derives an evaluation value representing the degree of improvement in blur in the applied image. Imaging device.

9. In the imaging apparatus according to claim 1, The processor executes a process to generate a depth map corresponding to the captured image based on the depth estimate of the subject portion corresponding to each partial image region in the captured image. Imaging device.

10. The processor, Encoded imaging processing involves controlling an imaging system including a mask that forms an encoding aperture, an optical system, and an image sensor so that encoded imaging is performed, and storing the image obtained by the encoded imaging in memory. A decoding image generation process is performed on the captured image for multiple point spreading functions, each with a different depth from the imaging system for the corresponding point light source, based on the point spreading function of the imaging system, and the multiple decoded images obtained are stored in the memory. An evaluation value calculation process which applies an evaluation function to each of the multiple decoded images and stores the obtained evaluation values ​​in the memory, Based on the aforementioned multiple evaluation values, a depth search process is performed to search for depth estimates of the subject portion corresponding to each partial image region in the captured image. Based on the aforementioned plurality of point spreading functions, for each of the plurality of point spreading function groups, each having a different range from the minimum to the maximum depth of the corresponding point light source, the decoded image generation process, the evaluation value calculation process, and the depth search process are executed sequentially. This includes performing a process to determine the depth estimate of the subject portion corresponding to each sub-image region in the captured image, based on the execution result of the depth search process corresponding to each of the plurality of point spread functions. A method for estimating the depth of a subject.

11. In the processor, Encoded imaging processing involves controlling an imaging system including a mask that forms an encoding aperture, an optical system, and an image sensor so that encoded imaging is performed, and storing the image obtained by the encoded imaging in memory. A decoding image generation process is performed on the captured image for multiple point spreading functions, each with a different depth from the imaging system for the corresponding point light source, based on the point spreading function of the imaging system, and the multiple decoded images obtained are stored in the memory. An evaluation value calculation process which applies an evaluation function to each of the multiple decoded images and stores the obtained evaluation values ​​in the memory, Based on the aforementioned multiple evaluation values, a depth search process is performed to search for depth estimates of the subject corresponding to each partial image region in the captured image. Based on the aforementioned multiple point spreading functions, the decoding image generation process, the evaluation value calculation process, and the depth search process are sequentially executed for each of the multiple point spreading function groups, each having a different range from the minimum to the maximum depth of the corresponding point light source. Based on the execution results of the depth search process corresponding to each of the aforementioned group of point spread functions, a process is executed to determine the depth estimate of the subject corresponding to each partial image region in the captured image. program.

Images