Imaging apparatus, depth map generation method, and program

Abstract

To improve practicality in subject depth estimation by coded imaging.SOLUTION: A calculation control unit 30 executes an imaging process that controls an encoding imaging system 10 to encode and image a subject 90 to obtain a first captured image P1, an image generation process that reduces the resolution of the first captured image P1 to obtain a second captured image P2, a partial area setting process that sets a partial area in the first captured image P1, a process that performs a first decoding process on a partial image PR1 corresponding to the partial area in the first captured image P1 to obtain a first depth map DM1, and a process that performs a second decoding process on the second captured image P2 to obtain a second depth map DM2.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an imaging device, a depth map generating method, and a program. [Background technology]

[0002] In the field of coded imaging, a technique called DFD (Depth From Defocus) is known. DFD is a technique that estimates the distance from the imaging system or optical system of an imaging device to a subject, i.e., the depth or perspective of the subject, based on the degree of edge blurring in an image obtained by imaging.

[0003] The DFD technique is described in, for example, Patent Document 1 or Non-Patent Document 1. In the DFD technique described in Non-Patent Document 1, a mask called a coded aperture, whose geometric pattern of the aperture through which light passes is known in advance, is prepared. Next, coded imaging is performed in which the prepared mask is placed in the light entrance area of ​​the optical system and an object is imaged. Next, a decoding process based on a point spread function specific to the mask used is performed on the captured image obtained by the coded imaging, and the depth of the object is estimated. Note that the point spread function is generally called a PSF (Point Spread Function), and is also called a blur function, blur spread function, point image distribution function, etc. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2011 / 158508 [Non-patent literature]

[0005] [Non-Patent Document 1] "Coded Aperture Pairs for Depth from Defocus and Defocus Deblurring" C. Zhou, S. Lin and SK Nayar, International Journal of Computer Vision, Vol. 93, No. 1, pp. 53, May. 2011. Summary of the Invention [Problem to be solved by the invention]

[0006] DFD technology is still in its infancy, and there is still room for improvement in terms of practicality. Due to the above circumstances, there is a demand for more practical DFD technology. [Means for solving the problem]

[0007] Among the inventions disclosed in this application, representative ones will be outlined as follows.

[0008] A representative embodiment of the present invention is an imaging device that includes a coding imaging system and an arithmetic and control unit, and the arithmetic and control unit executes an imaging process of controlling the coding imaging system to code and capture an image of a subject to obtain a first captured image, an image generation process of reducing the resolution of the first captured image to obtain a second captured image, a partial region setting process of setting a partial region in the first captured image, a process of performing a first decoding process on a partial image corresponding to the partial region in the first captured image to obtain a first depth map, and a process of performing a second decoding process on the second captured image to obtain a second depth map. , an imaging device.

[0009] A representative embodiment of the present invention is a depth map generation method that executes an imaging process of encoding and imaging a subject using an encoding imaging system to obtain a first captured image, an image generation process of reducing the resolution of the first captured image to obtain a second captured image, a partial region setting process of setting a partial region in the first captured image, a process of performing a first decoding process on a partial image corresponding to the partial region in the first captured image to obtain a first depth map, and a process of performing a second decoding process on the second captured image to obtain a second depth map.

[0010] A representative embodiment of the present invention is a program for causing a computer to execute an imaging process of encoding and imaging a subject using a coding imaging system to obtain a first captured image, an image generation process of reducing the resolution of the first captured image to obtain a second captured image, a partial region setting process of setting a partial region in the first captured image, a process of performing a first decoding process on a partial image corresponding to the partial region in the first captured image to obtain a first depth map, and a process of performing a second decoding process on the second captured image to obtain a second depth map. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a diagram showing an example of installation of the imaging device according to the first embodiment. [Diagram 2] FIG. 1 illustrates an example of a configuration of an imaging device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a configuration of an encoding imaging system. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of an arithmetic control unit. [Diagram 5] FIG. 2 is a diagram illustrating an example of a functional block configuration of an arithmetic and control unit. [Figure 6] FIG. 4 is a diagram showing an example of a processing flow in the imaging device according to the first embodiment. [Figure 7] FIG. 13 is a diagram showing an example of a captured image used in the decoding process of the reference method. [Figure 8]FIG. 13 is a diagram showing an example of an environment for executing a decoding operation and an estimated amount of calculation when a decoding process is performed by a reference method. [Figure 9] FIG. 2 is a diagram showing an example of a captured image used in the decoding process of the method according to the first embodiment. [Figure 10] FIG. 2 is a diagram showing an example of an environment for executing a decoding operation and an estimated value of a computation amount when the decoding process is performed by the method according to the first embodiment. [Figure 11] FIG. 11 is a diagram showing the estimated value of the amount of calculation in the reference method and the estimated value of the amount of calculation in the method according to the present embodiment side by side. [Figure 12] FIG. 11 is a diagram showing an estimate of the data amount of a depth map obtained by a reference method and an estimate of the data amount of a depth map obtained by a method according to the present embodiment side by side. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Before describing each embodiment of the present invention, one specific problem in the DFD technology that the present inventors have discovered will be described.

[0013] The state of blur of a subject in a captured image generally depends on a point spread function determined by the optical system of an imaging device, the shape of a light incidence region of the optical system, etc. When a mask that partially blocks light is installed in the light incidence region of the optical system, the point spread function is determined for each mask. Imaging a subject with an imaging device equipped with a mask is called coded imaging. When a subject is coded, a blurred image is obtained based on a point spread function specific to the mask used.

[0014] When the blurred image is subjected to a decoding process that performs deconvolution based on a point spread function specific to the mask used, a decoded image with improved or suppressed blur and depth information of objects corresponding to each position in the decoded image are obtained.

[0015] On the other hand, the inventors of the present invention have studied a DFD technique using coded imaging with a mask and have recognized that in order to ensure sufficient accuracy in estimating the depth of a subject, a large amount of calculation is required for the decoding process, and the calculation time is relatively long. In other words, the inventors of the present invention have found a problem in which, when estimating the depth of a subject by performing a decoding process on an image obtained by coded imaging, it is necessary to reduce the amount of calculation required for the decoding process and shorten the calculation time while ensuring the necessary estimation accuracy.

[0016] The present inventors have devised the present invention in order to solve the above problems. Hereinafter, an embodiment of the present invention will be described. Note that the embodiment described below is an example for carrying out the present invention, and does not limit the technical scope of the present invention. In the following embodiments, components having the same functions are given the same reference numerals, and repeated description thereof will be omitted unless particularly necessary.

[0017] (Embodiment 1) Overview of the imaging device according to the first embodiment An imaging device according to embodiment 1 of the present application will be described. The imaging device according to embodiment 1 of the present application is an imaging device including a coding imaging system and an arithmetic and control unit, and the arithmetic and control unit executes an imaging process for controlling the coding imaging system to code-image-capture an object and obtain a first captured image, an image generation process for reducing the resolution of the first captured image to obtain a second captured image, a partial region setting process for setting a partial region in the first captured image, a process for performing a first decoding process on a partial image corresponding to the partial region in the first captured image to obtain a first depth map, and a process for performing a second decoding process on the second captured image to obtain a second depth map.

[0018] That is, the imaging device according to the first embodiment is characterized in that, in estimating the depth (depth) of the entire subject based on encoded imaging, a highly accurate and computationally intensive decoding process is performed on some partial regions of the entire image region of the captured image that are considered to be of high importance, while a low accuracy and computationally intensive decoding process is performed on regions of the entire image region of the captured image that include other regions excluding the partial regions, thereby ensuring the necessary accuracy overall while reducing the overall computational complexity, thereby speeding up the decoding process or data transfer.

[0019] <Installation and configuration examples of imaging device> Fig. 1 is a diagram showing an example of installation of an imaging device according to embodiment 1. In embodiment 1, as shown in Fig. 1, imaging device 1 is installed in an automobile 100, which is an example of a vehicle. Imaging device 1 is installed so as to capture encoded images of a subject 90 ahead of automobile 100. The subject 90 includes a vehicle ahead 92, which is a vehicle traveling in front of automobile 100.

[0020] Fig. 2 is a diagram showing an example of the configuration of the imaging device 1. As shown in Fig. 2, the imaging device 1 has a coding imaging system 10, a calculation control unit 30, an operation unit 37, and a display unit 38. The coding imaging system 10, the operation unit 37, and the display unit 38 are each connected to the calculation control unit 30.

[0021] The encoding imaging system 10 is an imaging system having an angle of view α, which performs encoding imaging of a subject 90 based on a control signal from the arithmetic and control unit 30, and sends information on the obtained captured image, a high-resolution entire captured image P1, to the arithmetic and control unit 30. The high-resolution entire captured image P1 means a captured image that represents the entire image that has not been reduced in resolution.

[0022] The arithmetic and control unit 30 reduces the resolution of the high-resolution entire captured image P1 to generate a low-resolution entire captured image P2. The low-resolution entire captured image P2 means a captured image that represents the entire image with reduced resolution. The arithmetic and control unit 30 specifies a high-resolution partial captured image PR1 corresponding to a set partial region R1 in the image region of the high-resolution entire captured image P1. The partial region R1 is, for example, a region in the high-resolution entire captured image P1 where a depth estimation of the subject is likely to be obtained with high accuracy.

[0023] The calculation control unit 30 performs a first decoding process on the high-resolution partial captured image PR1 to obtain a high-precision partial decoded image QR1 with suppressed blurring and a depth estimate (depth) D1 of the subject at each position in the high-precision partial decoded image QR1, and generates a partial depth map DM1 based on the high-precision partial decoded image QR1 and the depth estimate D1. The calculation control unit 30 also performs a second decoding process on the low-resolution entire captured image P2 to obtain a low-precision entire decoded image Q2 with suppressed blurring and a depth estimate (depth) D2 of the subject at each position in the low-precision entire decoded image Q2, and generates an entire depth map DM2 based on the low-precision entire decoded image Q2 and the depth estimate D2.

[0024] The calculation control unit 30 generates a composite depth map DM3 by superimposing and synthesizing the partial depth map DM1 on the entire depth map DM2, and outputs the composite depth map DM3 to the external device 2.

[0025] The arithmetic control unit 30 accepts input of information related to commands, specifications, etc., based on the user's operation of the operation unit 37. The arithmetic control unit 30 also displays information for the user on the display unit 38. The operation unit 37 may be, for example, a mouse, a key, a button, a dial, etc. The display unit 38 may be, for example, a liquid crystal panel, an organic EL panel, etc. The operation unit 37 and the display unit 38 may be integrally configured, and may be, for example, a touch panel.

[0026] Here, the encoding imaging system 10 is an example of the "encoding imaging system" in the present application. The calculation control unit 30 is an example of the "calculation control unit" in the present application. The high-resolution entire captured image P1 is an example of the "first captured image" in the present application. The high-resolution partial captured image PR1 is an example of the "partial image" in the present application. The low-resolution entire captured image P2 is an example of the "second captured image" in the present application.

[0027] The configuration of the encoding imaging system 10 will now be described in more detail. Fig. 3 is a diagram showing an example of the configuration of the encoding imaging system 10. As shown in Fig. 3, the encoding imaging system 10 has an optical system 11, an imaging element 12, and a mask M1.

[0028] The optical system 11 is, for example, a wide-angle lens with a comparatively wide angle of view. The optical system 11 collects light L coming from a subject 90, and forms an image on the light receiving surface 12a of the image sensor 12.

[0029] The mask M1 is an optical filter that passes a portion of the light L entering the optical system 11 and blocks the other portion of the light L. The mask M1 is placed in a light entrance area where the light L enters the optical system 11 from the subject 90. The mask M1 is placed, for example, on the subject 90 side of the optical system 11. The mask M1 may be placed inside the optical system 11. The "mask" is also called a "coded aperture," "coded diaphragm," "aperture," etc.

[0030] The imaging element 12 is an electronic component that performs photoelectric conversion, and has a large number of photoelectric conversion elements that are two-dimensionally arranged. These many photoelectric conversion elements form a light receiving surface 12a. The imaging element 12 performs photoelectric conversion to convert the light and darkness of the image formed on the light receiving surface 12a into an amount of electric charge. When imaging is performed, which is a process of instantaneously capturing an electric signal obtained by photoelectric conversion, the imaging element 12 obtains and outputs data of a high-resolution entire captured image P1 that represents the subject 90. The imaging is performed, for example, by a rolling shutter method or a global shutter method.

[0031] The “imaging element” is also called an “image sensor.” The imaging element 12 is, for example, a CCD (Charge Coupled Devices) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0032] Light L arriving from a point image h1 on the surface 90a of the subject 90 passes through a mask M1, which is a coded aperture, and enters the optical system 11, and the light L passing through the optical system 11 forms a blurred image on the light receiving surface 12a of the image sensor 12. This blurred image is the point image h1 coded with a point spread function f1 specific to the coding imaging system 10 including the mask M1.

[0033] The configuration of the arithmetic control unit 30 will be described. Fig. 4 is a diagram showing an example of a hardware configuration of the arithmetic control unit 30. The arithmetic control unit 30 is, for example, a computer, and has a processor 31, a memory 32, an interface 33, and a communication bus 34 as shown in Fig. 4. The processor 31, the memory 32, and the interface 33 are connected to each other via the communication bus 34, and transmit and receive data.

[0034] The processor 31 is, for example, a central processing unit (CPU), a micro-processing unit (MPU), a microcontroller, or a graphics processing unit (GPU).

[0035] The memory 32 is, for example, a semiconductor storage device, and may include a magnetic disk, an optical disk, etc. The memory 32 stores a program PG and various data DT. The processor 31 functions to perform various processes by reading and executing the program PG.

[0036] The interface 33 is connected to the processor 31, the external device 2, etc., and mediates the transmission and reception of signals or data between these devices.

[0037] When the arithmetic control unit 30 is a computer, all or part of the computer may be configured with semiconductor circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a CPLD (Complex Programmable Logic Device), etc. The arithmetic control unit 30 may also include electronic circuits such as an image processing engine that processes data output from the imaging element 12.

[0038] <Function block configuration of the arithmetic and control unit> The following describes the functional block configuration of the arithmetic and control unit 30. Each functional block constituting the arithmetic and control unit 30 is realized by the processor 31 reading out and executing a program PG stored in the memory 32.

[0039] Fig. 5 is a diagram showing an example of a functional block configuration of the arithmetic control unit 30. As shown in Fig. 5, the arithmetic control unit 30 has, as functional blocks, a main control unit 301, an encoding imaging control unit 302, a captured image storage unit 303, a partial region setting unit 304, an image resolution conversion unit 305, a point spread function storage unit 306, a decoding processing unit 307, a depth map storage unit 308, and a data synthesis output unit 309.

[0040] The main control unit 301 controls each unit so that each process from encoding imaging to depth map output is carried out, performs various settings, or controls execution processing steps.

[0041] The encoding imaging control unit 302 controls the encoding imaging system 10 so as to perform encoding imaging of the subject 90. When the encoding imaging is performed, a high-resolution entire captured image P1 having a resolution substantially equal to that of the image sensor 12 is obtained and sent to the captured image storage unit 303.

[0042] The captured image storage unit 303 stores a high-resolution entire captured image P1, which is a captured image obtained by encoding imaging. The captured image storage unit 303 also stores a low-resolution entire captured image P2, which is obtained by reducing the resolution of the high-resolution entire captured image P1 by the image resolution conversion unit 305.

[0043] The partial region setting unit 304 sets a partial region R1 in the high-resolution entire captured image P1. The partial region R1 is set to a region where a high-precision, i.e., a high-resolution depth map in terms of space or time is likely to be required in, for example, an external device 2 that uses the depth map output from the imaging device 1.

[0044] The external device 2 is, for example, a device that provides driving assistance for a vehicle such as an automobile. In this case, the area near the center of the captured image is often considered important. This is because the area near the center of the captured image is likely to be a place where the vehicle on which the imaging device 1 is installed will travel or pass in the future, and it is considered that there is a constant need to estimate the depth of the preceding vehicle or obstacle with high accuracy. Therefore, in this embodiment, the partial area R1 is set to the center of the high-resolution entire captured image P1.

[0045] The partial region R1 may be set at a position designated by the user, or may be set based on the analysis result of the high-resolution whole image P1. For example, when the position of an object such as a vehicle ahead, a bicycle, a pedestrian, or a traffic light is specified based on the analysis result of the high-resolution whole image P1, the partial region R1 may be set at the position of the specified object. Furthermore, the partial region R1 may be multiple. Furthermore, the partial region R1 is assumed to be a rectangular region, but may be a circular, elliptical, or polygonal region.

[0046] The image resolution conversion unit 305 converts the resolution of the high-resolution entire captured image P1 obtained by the encoding imaging into a low-resolution entire captured image P2. The obtained low-resolution entire captured image P2 is sent to the captured image storage unit 303.

[0047] The point spread function storage unit 306 stores a plurality of types of point spread functions f1, ..., fj used in the first and second decoding processes. The point spread function f is a function that specifies what type of blur pattern B a point image h1 of a subject 90 located at a distance d from the encoding imaging system 10 produces on the light receiving surface 12a of the image sensor 12 when encoding imaging is performed using the mask M1 as the encoding aperture. In other words, the point spread function f can be said to be a function with the distance d as a parameter.

[0048] Let us assume that a point image hz of an object located at a distance dz is encoded and captured using a mask M1, an optical system 11, and an image sensor 12 to obtain a captured image Pz. If this captured image Pz is decoded using a point spread function fz having the distance dz as a parameter, a decoded image Qz without blur is obtained, and the distance of the object can be estimated to be dz.

[0049] That is, by performing decoding on a captured image in which the distance d to the subject 90 is unknown using multiple types of point spread functions f each having a different parameter distance d, and searching for the point spread function f in which blur is most improved, the distance d, which is a parameter of the searched point spread function f, can be estimated as the distance, i.e., depth, of the subject 90. Also, by performing the decoding process using more types of point spread functions f each having a different parameter distance d, it becomes possible to estimate the depth of the subject 90 with higher resolution and higher accuracy.

[0050] Based on various settings performed in advance, the decoding processing unit 307 performs a first decoding process on the high-resolution partial captured image PR1 and a second decoding process on the low-resolution overall captured image P2. The first decoding process is a decoding process that uses a first number n of relatively large or many types of point spread functions fa1, …, fan. Here, it will be referred to as a high-precision partial decoding process. Also, the second decoding process is a decoding process that uses a second number m of relatively small or few types of point spread functions fb1, …, fbm (m < n). Here, it will be referred to as a low-precision overall decoding process.

[0051] The decoding processing unit 307 executes a high-precision partial decoding process, which is the first decoding process, on the high-resolution partial captured image PR1 to obtain a high-precision partial decoded image QR1 with a high degree of blur suppression and a depth estimation value D1 of the subject 90 with high accuracy in the high-precision partial decoded image QR1. The decoding processing unit 307 further generates a high-precision partial depth map DM1 based on the high-precision partial decoded image QR1 and the depth estimation value D1 of the subject with high accuracy.

[0052] Also, the decoding processing unit 307 executes a low-precision overall decoding process, which is the second decoding process, on the low-resolution overall captured image P2 to obtain a low-precision overall decoded image Q2 with a low degree of blur suppression and a depth estimation value D2 of the subject 90 with low accuracy in the low-precision overall decoded image Q2. The decoding processing unit 307 further generates a low-precision overall depth map DM2 based on the low-precision overall decoded image Q2 and the depth estimation value D2 of the subject with low accuracy.

[0053] Note that the decoding processing unit 307 may perform the high-precision partial decoding process, which is the first decoding process, at a relatively high first frequency and perform the low-precision overall decoding process, which is the second decoding process, at a relatively low second frequency. The first frequency is basically the same as the frequency at which the encoded imaging is performed, but is not limited thereto.

[0054] The depth map storage unit 308 stores the depth map generated by the decoding processing unit 307. In this embodiment, the depth map storage unit 308 stores a high-precision partial depth map DM1 and a low-precision overall depth map DM2.

[0055] The data synthesis output unit 309 synthesizes depth maps stored in the depth map storage unit 308 to obtain a synthesized depth map, and outputs the synthesized depth map to the outside. More specifically, the data synthesis output unit 309 superimposes a high-precision partial depth map DM1 on a low-precision overall depth map DM2 to generate and output a synthesized depth map DM3 in which only the partial region R1 is high-precision and the other parts are low-precision. The synthesized depth map DM3 is generated every time encoded imaging, i.e., high-precision partial decoding processing, is performed.

[0056] However, if the low-precision full decoding process, which is the second decoding process, is not executed every time the encoded imaging, i.e., the high-precision partial decoding process, is performed, the data synthesis output unit 309 synthesizes the most recently obtained low-precision full depth map DM2 and the most recently obtained high-precision partial depth map DM1 to generate a synthesized depth map DM3. Alternatively, if the low-precision full decoding process and the high-precision partial decoding process are not performed at the same encoded imaging timing, the data synthesis output unit 309 may output only the high-precision partial depth map DM1 obtained by the high-precision partial decoding process.

[0057] Note that the partial depth map DM1 is an example of a "first depth map" in this application, the full depth map DM2 is an example of a "second depth map" in this application, and the composite depth map DM3 is an example of a "third depth map" in this application.

[0058] Decoding process and point spread function Here, the decoding process performed in this embodiment and the point spread function used in the decoding process will be described in more detail.

[0059] First, the relationship between the point spread function and the decoding process will be described. As described above, light L from a point image h1 of a subject 90 located at a distance d1 from the encoding imaging system 10 passes through a mask M1, which is a coding aperture, and the optical system 11 to be imaged on the light receiving surface 12a of the image sensor 12. In this case, assume that light L from a point image h2 located at a distance d2 from the encoding imaging system 10 passes through the mask M1 and the optical system 11 to appear as a blurred image with a blur pattern B2 on the light receiving surface 12a.

[0060] If there is no change in the state and positional relationship of the mask M1, the optical system 11, and the light receiving surface 12a, the point spread function of the system where a point distant by a distance d2 becomes a blurred image of the blur pattern B2 on the light receiving surface 12a is defined as, for example, f2. Suppose there is a captured image Ps obtained by encoding and capturing a point image hs of a subject whose distance d from the encoding imaging system 10 is unknown, using the same system. If a decoding operation is performed on this captured image Ps using the above point spread function f2 and an image Qs without blur is obtained, it is clear that the point image hs is located at a position distant by the distance d2 from the encoding imaging system 10.

[0061] That is, to obtain a depth map from a captured image obtained by encoding imaging, a decoding operation is repeatedly performed by applying multiple point spread functions fk with different parameter distance d to a blurred image centered on each position x in the captured image. Then, the parameter distance d of the point spread function fz when an image with the least blur is obtained can be estimated as the distance from the encoding imaging system 10 of the object corresponding to that position x, i.e., the depth or depth of the object.

[0062] Therefore, in this embodiment, the decoding process is a process in which decoding operations are repeatedly performed by applying multiple types of point spread functions fn with different parameter distance d to multiple positions in an image obtained by encoded imaging, and the distance dz, which is the parameter of the point spread function fz applied when the image with the least blur is obtained, is estimated as the distance (depth, depth) of the subject of the image at that position.

[0063] Next, the decoding process will be described. In this embodiment, as described above, two types of decoding processes are executed, namely, a high-precision partial decoding process which is the first decoding process, and a low-precision whole decoding process which is the second decoding process. The first decoding process is executed with relatively high precision on a high-resolution partial captured image PR1 corresponding to a set partial region R1 among the high-resolution whole captured image P1 obtained by the coding imaging. The second decoding process is executed with relatively low precision on the whole of a low-resolution whole captured image P2 obtained by lowering the resolution of the high-resolution whole captured image P1 obtained by the coding imaging.

[0064] The high-precision partial decoding process, which is the first decoding process, is basically executed every time the encoded imaging is performed. On the other hand, the low-precision whole decoding process, which is the second decoding process, does not have to be executed every time the encoded imaging is performed, and may be executed once every certain number of times of encoded imaging. That is, in the low-precision whole decoding process, the execution period may be lengthened, and the temporal resolution may be lowered compared to the high-precision partial decoding process. Also, in the low-precision whole decoding process, the number of types of point spread functions applied to the decoding operation may be reduced compared to the high-precision partial decoding process, and the depth of the image of the subject may be estimated more roughly. That is, in the low-precision whole decoding process, the resolution of the estimated depth of the subject may be lowered compared to the high-precision partial decoding process.

[0065] Considering the decoding process for the image of the same area, the decoding process with low accuracy requires less calculation amount than the decoding process with high accuracy, and the resulting data amount is also smaller. Therefore, the decoding process is performed with low accuracy in the whole area including the area in the captured image that does not require high accuracy decoding. On the other hand, the area in the captured image that requires high accuracy decoding is set as a partial area, and the decoding process is performed with high accuracy in the partial area. In this way, the information obtained by decoding can be reduced in calculation amount and data transfer amount while ensuring the required accuracy as a whole, and the time required for calculation and data transfer can be reduced.

[0066] Taking into account the contents of the above-mentioned decoding process, the main control unit 301 sets the partial area of ​​the image area of ​​the captured image that is to be subjected to the decoding process with high accuracy, the resolution of the low-resolution entire captured image, the execution period (execution frequency) of the low-resolution entire decoding process, the type of point spread function to be applied to the low-resolution entire decoding process, and the type of point spread function to be applied to the high-resolution partial decoding process.

[0067] <Processing flow of the imaging device> Next, a description will be given of a processing flow in the imaging device according to embodiment 1. Fig. 6 is a diagram showing an example of a processing flow in the imaging device 1 according to embodiment 1. The processing flow in the imaging device 1 is configured by processes of steps S1 to S9, for example, as shown in Fig. 6.

[0068] In step S1, initial settings are performed. Specifically, the main control unit 301 sets the degree of resolution reduction when generating a low-resolution entire captured image P2 from a high-resolution entire captured image P1. Here, as an example, a setting is made such that a 4×4 pixel area in the high-resolution entire captured image P1 corresponds to one pixel. The pixel value after the resolution reduction is a representative pixel value in the corresponding 4×4 pixel area, for example, the average, median, or weighted average of the pixel values.

[0069] The main control unit 301 also sets a repetition period of the low-resolution full decoding process, which is the second decoding process, i.e., the ratio of how many times the low-resolution full decoding process is executed for each encoded imaging. Here, as an example, the low-resolution full decoding process is set to be executed once for every four encoded imaging. The high-resolution partial decoding process is set to be executed each time encoded imaging is performed.

[0070] The main control unit 301 also sets the number of types of point spread functions to be applied to each of the high-resolution partial decoding process, which is the first decoding process, and the low-resolution whole decoding process, which is the second decoding process. Here, as an example, the high-resolution partial decoding process is set to use 25 types of point spread functions, and the low-resolution whole decoding process is set to use 5 types of point spread functions.

[0071] The 25 types of point spread functions used in the high-resolution partial decoding process are, for example, point spread functions with a parameter d of 10 types from 1m to 10m in increments of 1m, 10 types from 12m to 30m in increments of 2m, and 5 types from 35m to 55m in increments of 5m. The five types of point spread functions used in the low-resolution full decoding process are, for example, point spread functions with a parameter d of 5m, 10m, 20m, 35m, and 50m, respectively.

[0072] The above initial settings may be performed automatically based on predetermined information, or may be performed based on any information input by the user via the operation unit 37.

[0073] In step S2, coding imaging is performed. Specifically, the coding imaging control unit 302 controls the coding imaging system 10 so as to perform coding imaging. A high-resolution entire captured image P1 obtained by the coding imaging is stored in the captured image storage unit 303.

[0074] In step S3, a partial region R1 is set. Specifically, the main control unit 301 sets the partial region R1 in a part of the image region of the high-resolution entire captured image P1 where highly accurate subject depth estimation is considered to be necessary. Here, as an example, the partial region R1 is set in the center of the entire image region.

[0075] In step S4, a high-precision partial decoding process, which is a first decoding process, is executed. Specifically, the decoding processing unit 307 performs a decoding operation on the high-resolution partial captured image PR1 corresponding to the partial region R1 in the high-resolution entire captured image P1, using a relatively large number of point spread functions fa1, ..., fan determined in the initial setting. By executing such high-precision partial decoding process, a high-precision partial depth map DM1 is finally obtained. The obtained partial depth map DM1 is stored in the depth map storage unit 308.

[0076] In step S5, it is determined whether or not it is time to execute the low-precision full decoding process, which is the second decoding process. Specifically, the main control unit 301 determines whether or not to execute the low-precision full decoding process at the timing of the current encoded imaging based on the execution cycle set in step S1. If it is determined that the low-precision full decoding process is to be executed (S5: Yes), the main control unit 301 advances the processing step to the next step S6. On the other hand, if it is determined that the low-precision full decoding process is not to be executed, the processing step is skipped to step S7.

[0077] In step S6, a low-precision whole decoding process, which is a second decoding process, is executed. Specifically, the image resolution conversion unit 305 reduces the resolution of the high-resolution whole captured image P1 according to the initial setting, and generates a low-resolution whole captured image P2. Next, the decoding processing unit 307 performs a decoding operation on the generated low-resolution whole captured image P2 using a relatively small number of point spread functions fb1, ..., fbm (n>m≧1) determined in the initial setting. By executing such a low-precision whole decoding process, a low-precision whole depth map DM2 is finally obtained. The obtained whole depth map DM2 is stored in the depth map storage unit 308.

[0078] In step S7, the synthesis of the depth maps is performed. Specifically, the data synthesis output unit 309 reads out the most recently obtained partial depth map DM1 and the entire depth map DM2 stored in the depth map storage unit 308, and synthesizes the partial depth map DM1 on the entire depth map DM2 to obtain a synthesized depth map DM3. The obtained synthesized depth map DM3 is stored in the depth map storage unit 308.

[0079] In step S8, the combined depth map DM3 is output. Specifically, the data combination output unit 309 reads out the most recently obtained combined depth map DM3 from the depth map storage unit 308, and outputs it to the external device 2.

[0080] In step S9, it is determined whether or not to continue the process. Specifically, the main control unit 301 detects whether an event that does not require the process to be continued has occurred. For example, it detects whether an error has occurred in the imaging device 1, whether a command to forcibly stop the process has been input by the user, etc. If such an event that does not require the process to be continued has not occurred, the main control unit 301 determines to continue the process (S9: Yes) and returns the processing step to step S2. On the other hand, if such an event has occurred, the main control unit 301 determines not to continue the process (S9: No) and ends the process.

[0081] Although an example of the processing flow in the imaging device 1 has been described above, the order of the processing steps in the processing flow is not limited to this example. For example, the processing of step S4 and the processing of steps S5 and S6 may be interchanged.

[0082] <Amount of calculation and data transfer required for decoding process> Here, the amount of calculation required for the decoding process and the amount of data of the resulting depth map will be estimated for each of a reference method considered to be common in DFD technology and the method according to this embodiment, and then a comparison will be made between them.

[0083] Fig. 7 is a diagram showing an example of a captured image used in the decoding process of the reference method, and Fig. 8 is a diagram showing an example of an environment for executing a decoding operation and an estimated value of a computation amount when the decoding process is performed by the reference method.

[0084] When encoding imaging is performed by the imaging device 1 installed in the automobile 100, a high-resolution whole image P1 in which a leading vehicle 92 is positioned near the center, as shown in FIG. 7, for example, is directly obtained.

[0085] When performing a decoding process on this high-resolution entire captured image P1 using a standard method that is considered to be common, the environment and the estimated amount of calculations for performing the decoding operation can be considered, for example, as shown in Fig. 8. As shown in Fig. 8, for example, the calculation range is the entire image, the image area is 5 x 5 (relative value), the number of pixels per unit area is 4 x 4 (relative value), the number of types of point spread functions used in the calculation of one decoding process is 25, and the number of decoding processes performed per unit time is 60. When the decoding process is performed in such an environment, the amount of calculations required for the decoding process per unit time can be expressed as a value obtained by multiplying all of these numbers together, which is 600,000 (relative value) in this example.

[0086] Fig. 9 is a diagram showing an example of a captured image used in the decoding process of the method according to embodiment 1. Fig. 10 is a diagram showing an example of an environment for executing a decoding operation and an estimated value of a computation amount when the decoding process is performed by the method according to embodiment 1.

[0087] The method according to the present embodiment has two types of decoding processes, namely, a low-precision whole decoding process and a high-precision partial decoding process. The captured image used in the low-precision whole decoding process is a low-resolution whole captured image obtained by lowering the resolution of a high-resolution whole captured image directly obtained by encoding and capturing images by the imaging device 1. The low-resolution whole captured image is, for example, a low-resolution whole captured image P2 obtained by lowering the resolution of the high-resolution whole captured image P1 shown in Fig. 7, as shown in the upper part of Fig. 9. On the other hand, the captured image used in the high-precision partial decoding process is a high-resolution partial captured image PR1, which is an image of a partial region R1 set as a region considered to be of high importance in the high-resolution whole captured image P1.

[0088] When performing low-precision overall decoding processing on the low-resolution overall captured image P2, the environment and the estimated amount of calculations for performing the decoding calculation can be considered, for example, as shown in Fig. 10. As shown in the right column of the table in Fig. 10, for example, the calculation range is the entire image, the image area is 5 x 5 (relative value), the number of pixels per unit area is 1 x 1 (relative value), the number of types of point spread functions used in one decoding processing calculation is 5, and the number of decoding processing executed per unit time is 15. One pixel in the low-resolution overall captured image P2 corresponds to 4 x 4 pixels in the high-resolution overall captured image. When the decoding processing is performed in such an environment, the amount of calculations required for the decoding processing per unit time can be expressed as a value obtained by multiplying all of these numbers, which is 1,875 (relative value) in this example.

[0089] Furthermore, when performing high-precision partial decoding processing on the high-resolution partial captured image PR1, the environment and the estimated amount of calculations for performing the decoding calculation can be considered, for example, as shown in Fig. 10. As shown in the left column of the table in Fig. 10, for example, the calculation range is a partial image, the image area is 1 x 1 (relative value), the number of pixels per unit area is 4 x 4 (relative value), the number of types of point spread functions used in one decoding processing calculation is 25, and the number of decoding processing executed per unit time is 60. The area of ​​the partial image, i.e., the high-resolution partial captured image PR1, is 1 / 25 of the entire captured image. When the decoding processing is performed in such an environment, the amount of calculations required for the decoding processing per unit time can be expressed as a value obtained by multiplying all of these numbers, which is 24,000 (relative value) in this example.

[0090] The amount of calculations required for the decoding process per unit time in the method of this embodiment is the sum of the amount of calculations for the low-precision overall decoding process and the amount of calculations for the high-precision partial decoding process, which in this example is 25,875 (relative value).

[0091] Fig. 11 is a diagram showing the estimated amount of calculation in the reference method and the estimated amount of calculation in the method according to this embodiment side by side. As can be seen from Fig. 11, the amount of calculation in the method according to this embodiment can be reduced to about 4.3% of the amount of calculation in the reference method. That is, according to this embodiment, the calculation speed can be increased while ensuring necessary information in the decoding process.

[0092] 12 is a diagram showing the estimated value of the data amount of the depth map obtained by the reference method and the estimated value of the data amount of the depth map obtained by the method according to the present embodiment. The data amount of the depth map can be considered to be proportional to the number of pixels of the decoded image obtained by decoding. The environment for performing the decoding operation by the reference method is that the calculation range is the whole image, the image area is 5×5 (relative value), and the number of pixels per unit area is 4×4 (relative), so by multiplying all these values, the data amount can be estimated as 400 (relative value).

[0093] On the other hand, in the environment in which the decoding operation is performed by the method according to the present embodiment, the calculation range is a partial image and an entire image. When the calculation range is a partial image, the image area is 1×1 (relative value) and the number of pixels per unit area is 4×4 (relative value), so by multiplying all these values, the amount of data can be estimated as 16 (relative value). When the calculation range is an entire image, the image area is 5×5 (relative value) and the number of pixels per unit area is 1×1 (relative value), so by multiplying all these values, the amount of data can be estimated as 25 (relative value). In the method according to the present embodiment, these are synthesized to use a synthesized depth map, so the amount of data of the depth map handled in the imaging device 1 can be estimated as 41 (relative value) by adding up these values.

[0094] As can be seen from Fig. 12, the amount of depth map data in the method according to the present embodiment can be reduced to about 10% of the amount of depth map data in the reference method. That is, according to the present embodiment, the amount of depth map data transfer can be reduced, so that the data transfer speed can be increased while ensuring necessary information in the decoding process.

[0095] As described above, according to the present embodiment, when generating a depth map based on encoded imaging, it is possible to increase the calculation speed and data transfer speed while ensuring necessary information. As a result, for example, it is possible to increase the frame rate of encoded imaging, increase the resolution of the imaging element, and reduce the performance of the processor to reduce component costs.

[0096] (Embodiment 2) A depth map generating method according to the second embodiment of the present invention will be described. The depth map generating method according to the second embodiment is a depth map generating method that executes an imaging process of encoding and capturing an image of a subject using an encoding imaging system to obtain a first captured image, an image generating process of reducing the resolution of the first captured image to obtain a second captured image, a partial region setting process of setting a partial region in the first captured image, a process of performing a first decoding process on a partial image corresponding to the partial region in the first captured image to obtain a first depth map, and a process of performing a second decoding process on the second captured image to obtain a second depth map.

[0097] According to such a depth map generation method, it is possible to provide a more practical DFD technology, as in the first embodiment. More specifically, according to this depth map generation method, it is possible to ensure high accuracy of depth estimation of a part of a subject that is considered to be important, while suppressing the amount of calculation in the decoding process as a whole, and to realize high-speed calculation processing or high-speed data transfer speed. As a result, it is possible to improve the practicality of the depth estimation technology using DFD.

[0098] (Embodiment 3) A program according to the third embodiment of the present invention will be described. The program according to the third embodiment is a program for causing a computer to execute an imaging process for encoding and imaging an object using an encoding imaging system to obtain a first captured image, an image generation process for reducing the resolution of the first captured image to obtain a second captured image, a partial region setting process for setting a partial region in the first captured image, a process for performing a first decoding process on a partial image corresponding to the partial region in the first captured image to obtain a first depth map, and a process for performing a second decoding process on the second captured image to obtain a second depth map.

[0099] The program may be a program for causing a computer to function as the arithmetic control unit 30 included in the imaging device according to the embodiment 1. The program may also be a program for causing a computer to execute the subject depth estimation method according to the embodiment 2.

[0100] Note that a non-transitory tangible computer-readable recording medium on which the above program is recorded is also an embodiment of the present invention.

[0101] According to such a program, it is possible to provide a more practical DFD technology, as in the embodiment 1. More specifically, according to this program, by executing the program by a computer, it is possible to suppress the amount of calculations in the decoding process as a whole, and to realize high-speed calculation processing or high-speed data transfer speed, while ensuring high accuracy of depth estimation of a part considered to be important in a subject, as in the embodiment 1. As a result, it is possible to improve the practicality of the depth estimation technology using DFD.

[0102] Although various embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modified examples are included. Furthermore, the above-mentioned embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these belong to the scope of the present invention. Furthermore, the numerical values ​​and the like included in the text and figures are merely examples, and the effect of the present invention is not impaired even if different ones are used. [Explanation of symbols]

[0103] 1...imaging device, 2...external device, 10...encoding imaging system, 11...optical system, 12...imaging element, 30...arithmetic and control unit, 31...processor, 32...memory, 33...interface, 37...operation unit, 38...display unit, 90...subject, 92...forward vehicle, D1...low-precision overall depth information, D2...high-precision partial depth information, L...light, M1...mask, PG...program, P1...high-resolution overall captured image, P2...low-resolution overall captured image, R1...partial region, PR1...high-resolution partial captured image, QR1...high-precision partial decoded image, Q2...low-precision overall decoded image, DM1...partial depth map, DM2...overall depth map, DM3...synthetic depth map

Claims

1. The imaging system includes an encoding / imaging system and an arithmetic and control unit. The arithmetic and control unit is an imaging process for controlling the encoding imaging system to capture an image of a subject in an encoded form to obtain a first captured image; an image generating process for obtaining a second captured image by reducing the resolution of the first captured image; a partial region setting process for setting a partial region in the first captured image; a process of performing a first decoding process on a partial image corresponding to the partial region in the first captured image to obtain a first depth map; and performing a second decoding process on the second captured image to obtain a second depth map. Imaging device.

2. 2. The imaging device according to claim 1, The arithmetic and control unit is and superimposing the first depth map on the second depth map to obtain a third depth map. Imaging device.

3. 2. The imaging device according to claim 1, the first decoding process is a process of applying a first number of types of point spread functions; the second decoding process is a process of applying a second number of types of point spread functions, the second number being smaller than the first number. Imaging device.

4. 2. The imaging device according to claim 1, The imaging process is repeated multiple times, the first decoding process is performed with a first frequency; the second decoding process is performed at a second frequency that is lower than the first frequency. Imaging device.

5. 2. The imaging device according to claim 1, the arithmetic control unit executes a process of outputting, to an external device, the first depth map and the second depth map, or a third depth map in which the first depth map is superimposed on the second depth map; Imaging device.

6. 6. The imaging device according to claim 5, The external device is a device that provides driving assistance for a vehicle. Imaging device.

7. 2. The imaging device according to claim 1, The encoding imaging system is installed in a vehicle and captures encoded images of a front area of ​​the vehicle. Imaging device.

8. 2. The imaging device according to claim 1, The partial region is set in the center of the first captured image. Imaging device.

9. 2. The imaging device according to claim 1, The partial region is set at a position designated by a user. Imaging device.

10. 2. The imaging device according to claim 1, The partial region is set based on an analysis result of the first captured image. Imaging device.

11. an imaging process for capturing an image of a subject using an encoding imaging system to obtain a first captured image; an image generating process for obtaining a second captured image by reducing the resolution of the first captured image; a partial region setting process for setting a partial region in the first captured image; a process of performing a first decoding process on a partial image corresponding to the partial region in the first captured image to obtain a first depth map; and performing a second decoding process on the second captured image to obtain a second depth map. Depth map generation method.

12. On the computer, an imaging process for capturing an image of a subject using an encoding imaging system to obtain a first captured image; an image generating process for obtaining a second captured image by reducing the resolution of the first captured image; a partial region setting process for setting a partial region in the first captured image; a process of performing a first decoding process on a partial image corresponding to the partial region in the first captured image to obtain a first depth map; and a process of performing a second decoding process on the second captured image to obtain a second depth map.

Images