Imaging device

Abstract

To provide an imaging device which can take an image to be a material for generating a clear and bright WDoF image and a clear and bright HDR image.SOLUTION: An imaging device (1) includes: a first imaging element (13) for taking a first image; a second imaging element (14) for taking a second image, the second imaging element having the same size of pixels as those of the first imaging element (13) and having an optical distance different from that of the first imaging element (13); and a third imaging element (15) for taking a third image with a brightness different from that of the first image and the second image, the third imaging element having the different size of pixels from those of the second imaging element (14).SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging device.

Background Art

[0002] Conventionally, a technique for generating a Wide Depth of Field (WDoF) image with a deep depth of field or a High Dynamic Range (HDR) image with a large dynamic range by synthesizing a plurality of images has been known. And Patent Documents 1 to 3 disclose an imaging device including three imaging elements for the purpose of generating such an image by a single incidence of light.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, for the purpose of obtaining a sharp image, the resolution of an imaging element has been increasing. However, with the increase in resolution, the light reception sensitivity of the imaging element decreases. Therefore, there is a problem that the generated image becomes dark under the same imaging conditions.

[0005] The present invention has been made in view of the above circumstances, and an object thereof is to provide an imaging device capable of imaging an image that serves as a material for generating a sharp and bright WDoF image and HDR image.

Means for Solving the Problems

[0006] To solve the above problems, an imaging device according to the present invention includes a first imaging element that captures a first image, a second imaging element that has the same pixel size as the first imaging element and a different optical distance from the first imaging element and captures a second image, and a third imaging element that has a pixel size different from that of the second imaging element and captures a third image having a different brightness from the first image and the second image.

Effect of the Invention

[0007] According to the present invention, an imaging device capable of capturing images that serve as materials for generating clear and bright WDoF images and HDR images can be obtained. Note that problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the invention will be described with reference to the drawings. This embodiment contributes to "9. Build the foundation for industry and technological innovation" among the Sustainable Development Goals (SDGs) proposed by the United Nations by realizing a highly versatile camera device.

[0010] [First Embodiment] [Configuration of Imaging Device 1] FIG. 1 is a hardware configuration diagram of the imaging device 1. The imaging device 1 is a device that images a subject and processes (combines) imaging data. As shown in FIG. 1, the imaging device 1 according to the first embodiment includes an imaging unit 10 (imaging section) and an image processing unit 20 (image processing section). The imaging device 1 can be used in a wide range of fields such as the medical field and the industrial field. As a typical application of the imaging device 1, it can be considered to be mounted on a rigid endoscope system.

[0011] The rigid endoscope system is composed of an operation unit, a rigid endoscope, and a control unit. The operation unit is a part that is held and operated by a doctor. Also, the rigid endoscope is detachably attached to the operation unit and is connected to the control unit via a cable. Furthermore, the imaging unit 10 is mounted on the operation unit. The rigid endoscope is a part that is inserted into the body to irradiate the observation target with irradiation light and transmits the light reflected in the body to the operation unit. The imaging unit 10 photoelectrically converts the light transmitted from the rigid endoscope (hereinafter referred to as "incident light") to generate image data and transmits it to the control unit through the cable. The image processing unit 20 is mounted on the control unit. The image processing unit 20 processes the image data transmitted from the imaging unit 10 and outputs it (for example, displays, stores).

[0012] [Configuration of Imaging Unit 10] The imaging unit 10 is a camera unit that integrates an optical lens 11, a spectroscopic prism 12, and image sensors 13, 14, 15 (imaging elements). The optical lens 11 condenses the light outside the imaging unit 10 (the light transmitted from the rigid endoscope in the above example) and makes it incident on the spectroscopic prism 12. The spectroscopic prism 12 splits the incident light that has entered through the optical lens 11 at a predetermined spectroscopic ratio and outputs it to each of the image sensors 13, 14, 15.

[0013] Figure 2 is a diagram showing an example of the structure of the spectroscopic prism 12. Since the configuration of the spectroscopic prism 12 is already well-known as disclosed in Patent Documents 1 to 3, a detailed description will be omitted, but for example, it has the following configuration. The spectroscopic prism 12 is composed of a first prism 121, a second prism 122, and a third prism 123. The first prism 121 and the second prism 122 are provided with incident surfaces 121a, 122a, reflection surfaces 121b, 122b, and exit surfaces 121c, 122c. The third prism 123 is provided with an incident surface 123a and an exit surface 123c.

[0014] The incident surface 121a of the first prism 121 is arranged perpendicular to the optical axis of the incident light L (spectroscopic light L1 + L2 + L3). The image sensor 13 faces the exit surface 121c of the first prism 121. The image sensor 14 faces the exit surface 122c of the second prism 122. Further, the image sensor 15 faces the exit surface 123c of the third prism 123.

[0015] Also, the reflection surface 121b of the first prism 121 and the incident surface 122a of the second prism 122 form a joint surface A that is inclined with respect to the optical axis of the incident light L on the downstream side of the optical path of the incident light L from the incident surface 121a. Therefore, the incident light L that has entered the first prism 121 through the incident surface 121a is, at the joint surface A, partially (spectroscopic light L1) reflected and directed toward the incident surface 121a, and the rest (spectroscopic light L2 + L3) is transmitted and enters the second prism 122. The optical axis of the spectroscopic light L2 + L3 coincides with the optical axis of the incident light L.

[0016] Furthermore, the reflecting surface 122b of the second prism 122 and the incident surface 123a of the third prism 123 form a joint surface B that is inclined with respect to the optical axis of the spectral light L2+L3 on the downstream side of the optical path of the spectral light L2+L3 from the joint surface A. Therefore, the spectral light L2+L3 incident on the second prism 122 through the incident surface 122a is, at the joint surface B, partially (spectral light L2) reflected and directed toward the incident surface 122a, and the rest (spectral light L3) is transmitted and incident on the third prism 123.

[0017] And the spectral light L1 reflected by the joint surface A is totally reflected by the incident surface 121a and guided to the image sensor 13 through the exit surface 121c. Also, the spectral light L2 reflected by the joint surface B is totally reflected by the incident surface 122a and guided to the image sensor 14 through the exit surface 122c. Furthermore, the spectral light L3 incident on the third prism 123 through the incident surface 123a is guided to the image sensor 15 through the exit surface 123c.

[0018] That is, the spectroscopic prism 12 divides (spectroscopically) the incident light L into spectral lights L1, L2, and L3 and guides them to the image sensors 13, 14, and 15 respectively. Here, the spectroscopic ratios of the spectral lights L1, L2, and L3 can be appropriately adjusted by the densities of the first to third prisms 121 to 123, the inclinations of the joint surfaces A and B with respect to the optical axis of the incident light L, the film covering the joint surfaces A and B, etc. The spectroscopic ratios of the spectral lights L1, L2, and L3 will be described later with reference to FIGS. 3 and 4.

[0019] Also, by changing the thicknesses of the first prism 121, the second prism 122, and the third prism 123 respectively, the optical path lengths of the spectral lights L1, L2, and L3 can be adjusted. That is, the spectroscopic prism 12 can adjust the optical distances (focal lengths) of the image sensors 13, 14, and 15.

[0020] The spectral lights L1, L2, and L3 incident on the image sensors 13, 14, and 15 are part of the incident light L that is incident on the optical lens 11 at the same timing. Also, the spectral lights L1, L2, and L3 incident on the image sensors 13, 14, and 15 are lights in the same wavelength band (typically, the wavelength band of visible light). Furthermore, the imaging conditions (e.g., exposure time, aperture, etc.) of the image sensors 13, 14, and 15 are the same.

[0021] The image sensors 13, 14, and 15 photoelectrically convert the spectral lights L1, L2, and L3 output from the spectroscopic prism 12 to generate image data, and transmit the generated image data to the image processing unit 20 through a cable. The image sensors 13, 14, and 15 are, for example, CMOS (Complementary Metal - Oxide - Semiconductor) or CCD (Charge - Coupled Device).

[0022] Also, for the image sensors 13, 14, and 15, those capable of color photography having, for example, a Bayer array are used. That is, the image sensors 13, 14, and 15 include an R filter that transmits only R light, a G filter that transmits only G light, and a B filter that transmits only B light. And, in front of each of the plurality of pixels of the image sensors 13, 14, and 15, one of the R filter, G filter, and B filter is arranged. More specifically, among the plurality of pixels constituting each of the image sensors 13, 14, and 15, an R filter is arranged in front of the first pixel, a G filter is arranged in front of a second pixel different from the first pixel, and a B filter is arranged in front of a third pixel different from the first pixel and the second pixel. Thereby, each of the image sensors 13, 14, and 15 can generate a color image from the spectral lights L1, L2, and L3 in the same wavelength band incident from the spectroscopic prism 12.

[0023] The image sensor 13 is an example of the "first imaging device", the image sensor 14 is an example of the "second imaging device", and the image sensor 15 is an example of the "third imaging device". Also, the image captured by the image sensor 13 is an example of the "first image", the image captured by the image sensor 14 is an example of the "second image", and the image captured by the image sensor 15 is an example of the "third image".

[0024] The image sensors 13, 14, and 15 are an assembly of a plurality of pixels arranged in a matrix. The image sensors 13 and 14 are imaging devices with the same resolution. Also, the image sensor 15 is an imaging device with a lower resolution than the image sensors 13 and 14. In the first embodiment, the resolution of the image sensors 13 and 14 is 4k (for example, 3840×2160), and the resolution of the image sensor 15 is 2k (for example, 1920×1080) will be described. However, if the resolution of the image sensor 15 is lower than that of the image sensors 13 and 14, the combination of the resolutions of the image sensors 13, 14, and 15 is not limited to the above example. As another example, the resolution of the image sensors 13 and 14 may be 8k (for example, 7680×4320), and the resolution of the image sensor 15 may be 4k.

[0025] Also, the image sensors 13, 14, and 15 have the same area. Furthermore, the image sensors 13, 14, and 15 have the same aspect ratio. That is, the number of pixels of the image sensor 15 is less than that of the image sensors 13 and 14. Also, the size of one pixel of the image sensor 15 is larger than that of one pixel of the image sensors 13 and 14. On the other hand, the number of pixels of the image sensors 13 and 14 is the same, and the size of one pixel of the image sensors 13 and 14 is the same. However, if the size of one pixel of the image sensor 15 is larger than that of one pixel of the image sensors 13 and 14, the resolution, area, and aspect ratio are not limited to the above example.

[0026] Then, by making the size of one pixel of the image sensor 15 larger than that of one pixel of the image sensors 13 and 14, when the light amounts of the spectral lights L1, L2, and L3 are made the same, the light amount received by one pixel of the image sensor 15 becomes larger than the light amount received by one pixel of the image sensors 13 and 14. That is, if the spectral ratios of the spectral lights L1, L2, and L3 are 1:1:1, the third image captured by the image sensor 15 becomes brighter than the first image and the second image captured by the image sensors 13 and 14. On the other hand, when the brightnesses of the first image, the second image, and the third image are made the same, the spectral ratio of the spectral light L3 can be made smaller than the spectral ratios of the spectral lights L1 and L2.

[0027] Furthermore, the optical distances of the image sensors 13 and 14 (that is, the optical path lengths of the spectral lights L1 and L2 from the incident surface 121a to the image sensors 13 and 14 respectively) are different. That is, the positions where the first image and the second image are in focus are different within the image. On the other hand, the optical distances of the image sensors 14 and 15 (that is, the optical path lengths of the spectral lights L2 and L3 from the incident surface 121a to the image sensors 14 and 15 respectively) are the same. That is, the positions where the second image and the third image are in focus coincide within the image.

[0028] [Example of spectral ratio] With reference to FIGS. 3 and 4, an example of the spectral ratios of the spectral lights L1, L2, and L3 incident on the image sensors 13, 14, and 15 will be described. FIG. 3 is a diagram showing an example of the relationship between the spectral ratio and the sensor output. FIG. 4 is a diagram showing another example of the relationship between the spectral ratio and the sensor output. FIGS. 3 and 4 are diagrams showing an example in which sensors with higher resolution than the image sensor 15 are used for the image sensors 13 and 14. The "sensor output" refers to the value of the signal output by the image sensors 13, 14, and 15 after photoelectrically converting the spectral lights L1, L2, and L3. The sensor output is a value that has a positive correlation with the brightness of the image. More specifically, the higher the sensor output, the brighter the image tends to be.

[0029] In these examples, when light that has not been split by the spectroscopic prism 12 is incident, the sensor outputs of the 4K resolution image sensors 13 and 14 are set to 1, and the sensor output of the 2K resolution image sensor 15, whose pixel area is approximately four times that of the image sensors 13 and 14, is set to 4. Also, the spectroscopic ratio is set so that the ratio of the sensor outputs of the image sensors 14 and 15 is 16 times, so that the dynamic range expansion effect obtained by synthesizing the images of the image sensors 14 and 15 by the HDR synthesis process described later becomes 16 times in all examples.

[0030] As in Example 1 shown in FIG. 3, when the spectroscopic ratios of the spectroscopies L1, L2, and L3 are 64:64:1, the sensor outputs of the image sensors 13, 14, and 15 are 0.50:0.50:0.03. That is, Example 1 is an example of darkening the third image as compared with the first image and the second image. On the other hand, as in Comparative Example 1 shown in FIG. 3, when the image sensor 15 is replaced with 4K (that is, the size of one pixel of the image sensors 13, 14, and 15 is made the same), and the spectroscopic ratios of the spectroscopies L1, L2, and L3 are 16:16:1, the sensor outputs of the image sensors 13, 14, and 15 are 0.48:0.48:0.03.

[0031] That is, when the sensor output of the image sensor 15 is made the same (that is, the brightness of the third image is made the same) between the case where the image sensor 15 has a 2K resolution (Example 1) and the case where it has a 4K resolution (Comparative Example 1), the spectroscopic ratio to the image sensor 15 is smaller in Example 1 (=1 / 129) than in Comparative Example 1 (=1 / 33). That is, by setting the image sensor 15 to 2K resolution, the brightness of the third image can be maintained with a small spectroscopic ratio. Also, the sensor outputs of the image sensors 13 and 14 are higher in Example 1 (=0.50) than in Comparative Example 1 (=0.48). That is, by setting the image sensor 15 to 2K resolution, the brightness of the first and second images is improved while maintaining the brightness of the third image among the first to third images obtained by one incidence of light.

[0032] Also, as in Example 2 shown in FIG. 4, when the spectral ratios of spectrometers L1, L2, and L3 are set to 1:1:4, the sensor outputs of image sensors 13, 14, and 15 become 0.17:0.17:2.67. That is, Example 2 is an example of brightening the third image as compared with the first image and the second image. On the other hand, as in Comparative Example 2 shown in FIG. 4, when image sensor 15 is replaced with 4K (i.e., the size of one pixel of image sensors 13, 14, and 15 is made the same), and the spectral ratios of spectrometers L1, L2, and L3 are set to 1:1:16, the sensor outputs of image sensors 13, 14, and 15 become 0.06:0.06:0.89.

[0033] That is, when image sensor 15 has a 2K resolution (Example 2), the sensor outputs of all image sensors 13, 14, and 15 obtained by one incidence of light are higher than those in the case of a 4K resolution (Comparative Example 2). Therefore, by setting image sensor 15 to a 2K resolution, the brightness of all of the first image, the second image, and the third image obtained by one incidence of light is improved.

[0034] Note that the spectral ratios of spectrometers L1, L2, and L3 are not limited to the examples in FIGS. 3 and 4. However, the spectral ratios of spectrometers L1 and L2 are the same. That is, the first image and the second image have the same brightness. On the other hand, the third image has a brightness different from that of the first image and the second image.

[0035] [Configuration of Image Processing Unit 20] Returning to FIG. 1, image processing unit 20 is an integrated circuit product in which various electronic components are integrated on one chip formed of an integrated circuit. Image processing unit 20 includes memory 21, a CPU (Central Processing Unit) 22, and an ISP (Image Signal Processing) 23.

[0036] The memory 21 includes at least a RAM (Random Access Memory) and a ROM (Read Only Memory). The memory 21 temporarily holds programs and control data necessary for the execution of the operations of the image processing unit 20, and further data or information generated during the operations of each part of the image processing unit 20. The RAM is, for example, a work memory used during the operations of each part of the image processing unit 20. The ROM stores and holds in advance programs and control data for controlling each part of the image processing unit 20.

[0037] The CPU 22 is a processor that controls the overall operation of the image processing unit 20. The CPU 22 performs control processing for coordinating the operations of each part of the image processing unit 20, input / output processing of data between each part of the image processing unit 20, arithmetic processing of data, and storage processing of data. The CPU 22 operates according to the programs and control data stored in the memory 21. The CPU 22 uses the memory 21 during operation and transfers and temporarily stores the data generated or acquired by the CPU 22 in the memory 21.

[0038] The ISP 23 is a processor that executes various image processes performed within the image processing unit 20. The ISP 23 reads out image data from the memory 21 and performs various image processes using the read-out image data. The ISP 23 uses the memory 21 during operation and transfers and temporarily stores the data or information generated or acquired by the ISP 23 in the memory 21.

[0039] FIG. 5 is a functional block diagram of the image processing unit 20 according to the first embodiment. FIG. 6 is a transition diagram of the image processing by the image processing unit 20 according to the first embodiment. As shown in FIG. 5, the image processing unit 20 includes a WDoF image synthesis unit 31, an up-conversion processing unit 32, an HDR image synthesis unit 33, and a recombining unit 34. By the CPU 22 or the ISP 23 executing the programs stored in the memory 21, each functional block (31 to 34) shown in FIG. 6 is realized.

[0040] The WDoF image synthesizing unit 31 synthesizes the first image and the second image to generate a WDoF (Wide Depth of Field) image which is an example of the fourth image. The WDoF image is an image with a deeper depth of field than the first image and the second image. The WDoF image may also be called an EDoF (Extended Depth of Field) image. The WDoF image synthesizing unit 31 generates a WDoF image, for example, by extracting and synthesizing the in-focus portions from the first image and the second image respectively. Thereby, a WDoF image with a wide range in focus is generated. Since the process of generating a WDoF image is already well-known, a detailed description thereof is omitted.

[0041] The up-conversion processing unit 32 up-converts the third image with a 2k resolution to the third image with a 4k resolution. The up-conversion processing unit 32 generates the third image with a 4k resolution, for example, by interpolating pixels around each pixel of the third image with a 2k resolution. Since the process of up-converting the resolution of an image is already well-known, a detailed description thereof is omitted.

[0042] The HDR image synthesizing unit 33 synthesizes the second image and the third image with a 4k resolution to generate an HDR (High Dynamic Range) image which is an example of the fifth image. The HDR image is an image with a larger dynamic range than the second image and the third image. As an example, when using the spectral ratio of the first embodiment, the HDR image synthesizing unit 33 replaces the overexposed portion of the base second image with the corresponding portion of the third image with a 4k resolution. As another example, when using the spectral ratio of the second embodiment, the HDR image synthesizing unit 33 replaces the underexposed portion of the base second image with the corresponding portion of the third image with a 4k resolution. Thereby, an HDR image with overexposure and underexposure suppressed is generated. Since the process of generating an HDR image is already well-known, a detailed description thereof is omitted.

[0043] The recombining unit 34 combines the WDoF image and the HDR image to generate a WDoF&HDR image, which is an example of the sixth image. The WDoF&HDR image has a larger dynamic range than the WDoF image and a deeper depth of field than the HDR image. As an example, the recombining unit 34 may process (e.g., interpolate, correct, or replace) the overexposed or underexposed portions in the base WDoF image with the HDR image. As another example, the recombining unit 34 may process (e.g., interpolate, correct, or replace) the out-of-focus portions in the base HDR image with the WDoF image. However, the specific process for generating the WDoF&HDR image is not limited to the foregoing examples.

[0044] Then, the image processing unit 20 outputs at least one of the generated WDoF image, HDR image, and WDoF&HDR image. As a specific example of the output, for example, it may be displayed on a display or stored in an external memory. Also, when the WDoF&HDR image is not output, the recombining unit 34 can be omitted.

[0045] [Advantages of the First Embodiment] According to the first embodiment, by increasing the resolution of the image sensors 13 and 14, sharp WDoF images and HDR images can be obtained. Also, by making the size of one pixel of the image sensor 15 larger than that of one pixel of the image sensors 13 and 14, the brightness of the first, second, and third images can be increased. As a result, the first to third images, which serve as materials for generating sharp and bright WDoF images and HDR images, can be obtained. Also, as another problem when using high-resolution ones for all three image sensors 13, 14, and 15, the power consumption increases and the heat generation of the imaging unit 10 increases. Therefore, by reducing the resolution of the image sensor 13, the power consumption of the imaging unit 10 is reduced, and thus this problem can also be solved.

[0046] Further, according to the first embodiment, by making the spectral ratio of spectral L3 smaller than that of spectral L1 and L2 as in Example 1, the whitewashed portion of the second image on the bright side is replaced with the corresponding portion of the third image on the dark side, and an HDR image with high resolution in the high-luminance region (i.e., bright region) can be generated. Such a setting of the spectral ratio is particularly effective when the observation target is a high-luminance region of the HDR image.

[0047] Also, according to the first embodiment, by making the spectral ratio of spectral L3 larger than that of spectral L1 and L2 as in Example 2, the blacked-out portion of the second image on the dark side is replaced with the corresponding portion of the third image on the bright side, and an HDR image with high resolution in the low-luminance region (i.e., dark region) can be generated. Such a setting of the spectral ratio is particularly effective when the observation target is a low-luminance region of the HDR image.

[0048] Furthermore, according to the first embodiment, by synthesizing the WDoF image and the HDR image to generate a WDoF&HDR image, an image with a deep depth of field and a large dynamic range can be obtained.

[0049] [Second Embodiment] Referring to FIGS. 7 to 9, the imaging device 1 according to the second embodiment will be described. Note that a detailed description of the common points with the first embodiment will be omitted, and the description will focus on the differences. The basic configuration of the imaging device 1 according to the second embodiment is common to the first embodiment. On the other hand, the second embodiment is different from the first embodiment in that the image sensors 13 and 14 have a 2k resolution and the image sensor 15 has a 4k resolution.

[0050] That is, the image sensor 15 (third imaging element) according to the second embodiment is an imaging element with a higher resolution than the image sensor 13 (first imaging element) and the image sensor 14 (second imaging element). Also, in the second embodiment, the number of pixels of the image sensor 15 is larger than that of the image sensors 13 and 14. Further, in the second embodiment, the size of one pixel of the image sensor 15 is smaller than that of one pixel of the image sensors 13 and 14. That is, in the present invention, the size of one pixel of the image sensor 15 only needs to be different from that of one pixel of the image sensors 13 and 14.

[0051] By making the size of one pixel of the image sensors 13 and 14 larger than that of one pixel of the image sensor 15, when the light amounts of the spectral lights L1, L2, and L3 are made the same, the light amount received by one pixel of the image sensors 13 and 14 becomes larger than the light amount received by one pixel of the image sensor 15. That is, if the spectral ratios of the spectral lights L1, L2, and L3 are 1:1:1, the first image and the second image captured by the image sensors 13 and 14 become brighter than the third image captured by the image sensor 15. On the other hand, when the brightnesses of the first image, the second image, and the third image are made the same, the spectral ratio of the spectral light L3 can be made larger than the spectral ratios of the spectral lights L1 and L2.

[0052] FIG. 7 is a diagram showing still another example of the relationship between the spectral ratio and the sensor output. FIG. 7 is a diagram showing an example in which sensors with a lower resolution than the image sensor 15 are used for the image sensors 13 and 14. As in Example 3 shown in FIG. 7, when the spectral ratios of the spectral lights L1, L2, and L3 are 4:4:1, the sensor outputs of the image sensors 13, 14, and 15 are 1.78:1.78:0.11. On the other hand, as in Comparative Example 3 shown in FIG. 7, when the image sensors 13 and 14 are replaced with 4K (that is, the sizes of one pixel of the image sensors 13, 14, and 15 are made the same) and the spectral ratios of the spectral lights L1, L2, and L3 are 16:16:1, the sensor outputs of the image sensors 13, 14, and 15 are 0.48:0.48:0.03.

[0053] That is, when the image sensors 13 and 14 are set to 2k resolution (Example 3), the sensor outputs of all the image sensors 13, 14, and 15 obtained by one incidence of light are higher compared to the case where the resolution is 4k (Comparative Example 3). Therefore, by setting the image sensors 13 and 14 to 2k resolution, the brightness of all of the first image, the second image, and the third image obtained by one incidence of light is improved.

[0054] FIG. 8 is a functional block diagram of the image processing unit 20 according to the second embodiment. FIG. 9 is a transition diagram of the image processing by the image processing unit 20 according to the second embodiment. The processes executed by each processing unit (31 to 34) constituting the image processing unit 20 are common to the first embodiment and the second embodiment. On the other hand, the images input to and output from each processing unit (31 to 34) are different between the first embodiment and the second embodiment.

[0055] First, the up-conversion processing unit 32 according to the second embodiment up-converts the first image with a 2k resolution to a 4k resolution and outputs it to the WDoF image synthesis unit 31. Further, the up-conversion processing unit 32 according to the second embodiment up-converts the second image with a 2k resolution to a 4k resolution and outputs it to the WDoF image synthesis unit 31 and the HDR image synthesis unit 33.

[0056] Further, the WDoF image synthesis unit 31 according to the second embodiment synthesizes the first image and the second image with a 4k resolution up-converted by the up-conversion processing unit 32 to generate a WDoF image (fourth image) with a 4k resolution. Furthermore, the HDR image synthesis unit 33 according to the second embodiment synthesizes the second image with a 4k resolution up-converted by the up-conversion processing unit 32 and the third image with a 4k resolution generated by the image sensor 15 to generate an HDR image (fifth image) with a 4k resolution.

[0057] However, the WDoF image synthesizing unit 31 according to the second embodiment may synthesize the first image with a 2k resolution and the second image with a 2k resolution to generate a WDoF image (fourth image) with a 2k resolution. Then, the up-conversion processing unit 32 according to the second embodiment may up-convert the 2k-resolution WDoF image generated by the WDoF image synthesizing unit 31 to generate a 4k-resolution WDoF image.

[0058] [Operation and Effect of Second Embodiment] According to the second embodiment, by making the size of one pixel of the image sensors 13 and 14 larger than that of one pixel of the image sensor 15, the first image, the second image, and the third image can be made brighter than in the first embodiment. Further, by setting the image sensor 15 to a 4k resolution, a sharper HDR image can be obtained as compared with the case where all of the image sensors 13, 14, and 15 are set to a 2k resolution.

[0059] [Other Embodiments] Note that part or all of each means realized by a program can also be realized by hardware such as an integrated circuit. Further, the program may be recorded on a non-transitory computer-readable recording medium and provided. The recording medium refers to, for example, a hard disk, an SD card, a DVD, or a server on the Internet.

[0060] As described above, various embodiments have been described with reference to the drawings. Needless to say, the present disclosure is not limited to such examples. It is obvious that those skilled in the art can conceive of various modification examples, correction examples, substitution examples, addition examples, deletion examples, and equivalent examples within the scope described in the claims, and it is naturally understood that those also belong to the technical scope of the present disclosure. Further, within the scope not departing from the gist of the invention, the components in the above-described various embodiments may be arbitrarily combined. [Industrial Applicability]

[0061] The present disclosure is useful as an imaging device capable of imaging an image serving as a material for generating a sharp and bright WDoF image and HDR image.

Explanation of Signs

[0062] 1 Imaging device 10 Imaging unit 11 Optical lens 12 Spectroscopic prism 13, 14, 15 Image sensor 20 Image processing unit 21 Memory 22 CPU 31 WDoF image synthesis unit 32 Up-conversion processing unit 33 HDR image synthesis unit 34 Re-synthesis unit 121 First prism 121a, 122a, 123a Incident surface 121b, 122b Reflecting surface 121c, 122c, 123c Exit surface 122 Second prism 123 Third prism

Claims

1. A first imaging element that captures a first image, A second imaging element that has the same pixel size as the first imaging element and a different optical distance from the first imaging element, and captures a second image, An imaging device comprising a third imaging element that has a pixel size different from that of the second imaging element and captures a third image having a brightness different from that of the first image and the second image.

2. The imaging device according to claim 1, wherein the third imaging element has a pixel size larger than that of the second imaging element.

3. The imaging device according to claim 1, wherein the first imaging element, the second imaging element, and the third imaging element are imaging elements in a Bayer array.

4. The imaging device according to claim 1, wherein the first imaging element, the second imaging element, and the third imaging element have the same area.

5. Comprising a spectroscopic prism that splits incident light and guides it to each of the first imaging element, the second imaging element, and the third imaging element, The imaging device according to claim 1, wherein the spectroscopic ratio of the spectroscopic prism with respect to the third imaging element is smaller than the spectroscopic ratios of the first imaging element and the second imaging element, respectively.

6. Comprising a spectroscopic prism that splits incident light and guides it to each of the first imaging element, the second imaging element, and the third imaging element, The imaging device according to claim 1, wherein the spectroscopic ratio of the spectroscopic prism with respect to the third imaging element is larger than the spectroscopic ratios of the first imaging element and the second imaging element, respectively.

7. The imaging device according to claim 5 or 6, wherein the spectroscopic ratios of the spectroscopic prism with respect to the first imaging element and the second imaging element are the same.

8. Comprising a spectroscopic prism that splits incident light into three spectroscopies in the same wavelength band and guides it to each of the first imaging element, the second imaging element, and the third imaging element, The imaging device according to claim 1, wherein each of the first imaging element, the second imaging element, and the third imaging element receives the spectroscopy in the same wavelength band and generates an image.

9. Comprising an image processing unit that processes the first image, the second image, and the third image, The image processing unit Combines the first image and the second image to generate a fourth image having a deeper depth of field than the first image and the second image, The imaging device according to claim 1, wherein the second image and the third image are combined to generate a fifth image having a larger dynamic range than the second image and the third image.

10. The imaging device according to claim 9, wherein the image processing unit generates a sixth image by synthesizing the fourth image and the fifth image.

Images