Imaging device

The dual-sensor imaging device with SPAD and CMOS sensors optimizes power usage and image quality by switching modes, addressing high-brightness issues and enhancing dynamic range through error correction and dynamic range expansion.

JP7591095B2Active Publication Date: 2024-11-27CANON KK
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2023091030
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-08-08
Filing Date
2023-06-01
Publication Date
2024-11-27
Estimated Expiration
2039-06-26

AI Technical Summary

Technical Problem

Existing imaging devices using avalanche photodiodes in Geiger mode face issues with high power consumption and photon counting errors, especially when capturing high-brightness subjects, due to the avalanche phenomenon and dead time effects.

Method used

The imaging device employs a dual-sensor system comprising a SPAD-type image sensor for still images and a CMOS-type image sensor for video, switching between them based on mode selection to optimize power consumption and image quality, and incorporates dynamic range expansion and error correction techniques to enhance image quality.

Benefits of technology

This approach improves image quality by leveraging the strengths of both SPAD and CMOS sensors, reducing power consumption and minimizing counting errors, enabling high-quality still images and videos with expanded dynamic range.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007591095000001
    Figure 0007591095000001
  • Figure 0007591095000002
    Figure 0007591095000002
  • Figure 0007591095000003
    Figure 0007591095000003
Patent Text Reader

Abstract

To provide an imaging apparatus that improves image quality by taking advantage of a SPAD imaging element and a CMOS imaging element.SOLUTION: An imaging apparatus 1 has: a first imaging element 103 that counts the number of incident photons and has a plurality of pixels outputting the counted values as first image signals; a second imaging element 123 that has a plurality of pixels outputting electrical signals according to the amount of electric charges obtained by photoelectrically converting incident light as second image signals; a second imaging signal processing unit 124 that acquires a luminance value; and a control unit 13 being creation means that selects any one of the first image signals and the second image signals and creates an image. When the luminance value is equal to or more than a predetermined threshold, the control unit selects the second image signals, and when the luminance value is less than the threshold, selects the first image signals.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an imaging device equipped with an imaging element using an avalanche photodiode. [Background technology]

[0002] Conventionally, charge accumulation type image sensors have been commonly used in digital cameras, video cameras, etc. The charge accumulation type is a method in which the light incident on a photodiode (PD) within a certain period of time is captured as an analog quantity called a voltage value.

[0003] In the charge accumulation method, when light is incident on the PD of each pixel, the PD generates and accumulates an electric charge in an almost linear manner according to the amount of incident light. The electric charge accumulated in the PD is transferred to the floating diffusion (FD), converted to a voltage, and amplified by the source follower (SF). The voltage output from each pixel is converted to a digital signal by an AD converter and then output outside the image sensor.

[0004] In the case of the charge accumulation method described above, for example, when the voltage of the FD is amplified by the SF, it is known that the S / N ratio deteriorates due to RTS (Random Telegraph Signal) noise generated at the gate interface of the SF.

[0005] Meanwhile, in recent years, there has been research into photon-counting image sensors that can measure the number of incident photons themselves and treat the incident light as a digital value by utilizing the avalanche phenomenon that occurs when an avalanche photodiode (APD) is operated in Geiger mode.

[0006] When an APD is operated in Geiger mode, for example, when one photon is incident on the APD, an observable level of current is generated due to the avalanche phenomenon. By converting this current into pulses and counting the number of pulses, it becomes possible to directly measure the number of incident photons, so no RTS noise is generated and it is expected that the S / N ratio will be improved. As an example of a sensing device using an APD, Patent Document 1 discloses a distance measurement sensor consisting of an APD with multiple pixels.

[0007] In Geiger mode, the avalanche phenomenon can be caused even by the incidence of a single photon, so it is also called SPAD (Single Photon Avalanche Diode). [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2014-81253 A Summary of the Invention [Problem to be solved by the invention]

[0009] In order to operate an APD in Geiger mode, a high electric field above the breakdown voltage must be applied. In addition, when photons are incident, a large current flows due to the avalanche phenomenon, which poses a problem of large power consumption.

[0010] Furthermore, when capturing an image of a high-brightness subject, multiple photons are incident in a short period of time (dead time), which creates the problem that photons cannot be counted properly if another photon is incident during the avalanche phenomenon caused by a single photon.

[0011] The present invention has been made in view of the above problems, and has an object to improve image quality by utilizing the advantages of a photon-counting type imaging element such as a SPAD and a charge-accumulation type imaging element such as a CMOS. [Means for solving the problem]

[0012] In order to achieve the above object, the imaging device of the present invention counts the number of incident photons and converts the counted value into a first color A first image sensor having a plurality of pixels that outputs an image signal, and a second image sensor that converts incident light into an electric signal according to the amount of charge obtained by photoelectric conversion. color a second image sensor having a plurality of pixels which are output as an image signal; an acquisition means for acquiring a luminance value; color Image Signal or The second color Image Signal Based on color A generating means for generating an image; ,before When the brightness value is equal to or greater than a predetermined threshold value, The generating means The second color Image Signal Generate color images based on If the threshold value is less than the threshold value, The generating means The first color Image Signal A control means for controlling the generation of a color image based on the Have do. Effect of the Invention

[0013] According to the present invention, image quality can be improved by utilizing the advantages of a photon-counting type imaging element such as a SPAD and a charge-accumulation type imaging element such as a CMOS. [Brief description of the drawings]

[0014] [Figure 1] FIG. 2A is a diagram showing an example of the appearance of an imaging device according to an embodiment of the present invention, and FIGS. 2B and 2C are a perspective view and a front view showing an example of the appearance of a camera module. [Diagram 2] 1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment. [Diagram 3] 1A and 1B are diagrams showing the configuration of a SPAD type imaging element and the circuit configuration of a pixel according to an embodiment. [Figure 4] 2A to 2C are diagrams for explaining the operation principle of a SPAD type imaging element according to an embodiment. [Diagram 5]FIG. 2 is a diagram showing the configuration of a CMOS type image sensor and the circuit configuration of a pixel according to an embodiment. [Figure 6] 5 is a flowchart illustrating an imaging operation according to the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of the appearance of an imaging device according to a modified example of the first embodiment. [Figure 8] FIG. 4 is a block diagram showing the configuration of an imaging device according to a modified example of the first embodiment. [Figure 9] 10 is a flowchart illustrating an imaging operation according to a modified example of the first embodiment. [Figure 10] 1A and 1B are diagrams for explaining count errors in a SPAD type imaging element. [Figure 11] FIG. 13 is a diagram showing an example of an image in which a count error occurs in a SPAD type imaging element. [Figure 12] 10 is a flowchart illustrating an imaging operation according to a second embodiment. [Figure 13] 13A to 13C are views for explaining a dynamic range expansion process according to the third embodiment. [Figure 14] 10 is a flowchart illustrating an imaging operation according to a third embodiment. [Figure 15] 13 is a flowchart illustrating an imaging operation according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The best mode for carrying out the present invention will be described in detail below with reference to the accompanying drawings. However, the dimensions, materials, shapes, and relative positions of the components exemplified in this embodiment should be appropriately changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to these examples.

[0016] <First embodiment> A first embodiment of the present invention will now be described with reference to Figures 1 to 6. Note that the imaging device of this embodiment is applicable to electronic still cameras and video cameras with movie function.

[0017] Fig. 1(a) is a diagram showing an example of the appearance of an imaging device according to an embodiment of the present invention. As shown in Fig. 1(a), the imaging device 1 of this embodiment has two optical units, a first imaging system 11 and a second imaging system 12.

[0018] 2 is a block diagram showing the configuration of the imaging device 1. The imaging device 1 includes a first imaging system 11, a first imaging signal processing unit 115, a first compression / decompression unit 116, a second imaging system 12, a second imaging signal processing unit 124, and a second compression / decompression unit 125. It also includes a control unit 13, an operation unit 14, an image display unit 15, and an image recording unit 16.

[0019] The first imaging system 11 is composed of a first optical barrel 101 (first imaging optical system), a shutter mechanism section 102, and a first imaging element 103. The first optical barrel 101 includes a lens for collecting light from a subject on the first imaging element 103, and a first optical mechanism section 111 equipped with an optical mechanism for adjusting the focus of the collected light, changing the optical imaging magnification, and adjusting the amount of light.

[0020] The first optical mechanism unit 111 is driven based on a control signal from the control unit 13. The shutter mechanism unit 102 is configured between the first optical barrel 101 and the first imaging element 103, and controls the exposure time for exposing the first imaging element 103 to light that has passed through the first optical barrel 101, in accordance with the control signal from the control unit 13.

[0021] The first imaging element 103 is a SPAD type imaging element made up of pixels using APDs, and performs imaging operation in response to a control signal from the control unit 13, and outputs an image signal that is a count value of the number of incident photons. The configuration of the first imaging element 103 will be described later.

[0022] The first imaging signal processing unit 115 performs image processing such as color correction processing, AE (Auto Exposure) processing, white balance processing, and optical shading correction processing on the image signal from the first imaging element 103 under the control of the control unit 13, and then outputs the image signal and a control signal to the control unit 13. The image signal and control signal output from the first imaging signal processing unit 115 are recorded in a RAM (not shown) configured in the control unit 13.

[0023] The first compression / expansion unit 116 operates under the control of the control unit 13, and performs compression / encoding processing in a predetermined data format such as the JPEG format on the image signal recorded in the RAM in the control unit 13 from the first imaging signal processing unit 115. It also performs decompression / decoding processing on encoded data of still images supplied from the control unit 13. Furthermore, it may be possible to perform compression / encoding / decompression / decoding processing of moving images using the MPEG (Moving Picture Experts Group) format or the like.

[0024] The second imaging system 12 is composed of a second optical barrel 121 (a second imaging optical system) and a second imaging element 123. The second optical barrel 121 has a similar configuration to the first optical barrel 101, and receives a control signal from the control unit 13 and performs optical control in a second optical mechanism unit 122.

[0025] The second imaging element 123 is a CMOS type imaging element, which reads out image signals from CMOS pixels by an XY readout method, which will be described later, and outputs the image signals in response to control signals from the control unit 13 .

[0026] The second imaging signal processing unit 124 performs image processing on the image signal from the second imaging element 123 in the same manner as the first imaging signal processing unit 115, and outputs an image signal and a control signal to the control unit 13. The image signal and control signal output from the second imaging signal processing unit 124 are recorded in a RAM (not shown) configured in the control unit 13.

[0027] The second compression / decompression section 125 performs the same processing as the first compression / decompression section 116 on the image signal processed by the second imaging signal processing section 124 .

[0028] In the present embodiment, the first imaging signal processing unit 115 and the second imaging signal processing unit 124, and the first compression / expansion unit 116 and the second compression / expansion unit 125 are configured separately. However, the present invention is not limited to this, and a configuration in which a single imaging signal processing unit and compression / expansion unit are used to process image signals obtained from the first imaging system 11 and the second imaging system 12 may be used.

[0029] The control unit 13 is, for example, a microcontroller including a CPU, a ROM, a RAM, and the like, and performs overall control of each unit of the imaging device 1 by executing a program stored in the ROM or the like.

[0030] The operation unit 14 is composed of various operation keys such as a shutter release button, levers, dials, etc., and outputs control signals according to input operations by the user to the control unit 13. The image display unit 15 is composed of a display device such as an LCD and an interface circuit for this, and generates an image signal to be displayed on the display device from the image signal supplied from the control unit 13, and supplies this signal to the display device to display the image.

[0031] The image recording unit 16 receives image data files encoded by the first compression / decompression unit 116 and the second compression / decompression unit 125 from the control unit 13 and stores them in a storage medium such as a portable semiconductor memory, an optical disk, a HDD, or a magnetic tape. It also reads out specified data from the storage medium based on a control signal from the control unit 13 and outputs it to the control unit 13.

[0032] 1(b) and 1(c) are diagrams showing an example of the external configuration of a camera module equipped with an imaging system that can be used in the imaging device 1 of this embodiment. Fig. 1B is a perspective view of the camera module 200, and Fig. 1C is a front view of the camera module 200.

[0033] The camera module in this embodiment is a camera module in which two imaging elements are mounted by connecting two imaging systems. Note that the module may be called by other names such as a package.

[0034] The camera module 200 is configured by fixing the first imaging system 11 and the second imaging system 12 with a connecting member 400 having a rectangular plate-like shape. The first imaging system 11 is equipped with a first optical barrel 101 (lens unit) including a first optical mechanism section 111, a shutter mechanism section 102, a first imaging element 103, etc. The second imaging system 12 is configured by mounting a second optical barrel 121 (lens unit) including a second optical mechanism section 122, a second imaging element 123, etc., similar to the first imaging system 11.

[0035] The connecting member 400 has a rectangular plate-like shape with an outline larger than the size in the planar direction when the lens unit of the first imaging system 11 and the lens unit of the second imaging system 12 are arranged side by side. Furthermore, a rectangular insertion hole portion into which the lens unit of the first imaging system 11 is inserted and a rectangular insertion hole portion into which the lens unit of the second imaging system 12 is inserted are symmetrically formed through the connecting member 400. The lens unit of the first imaging system 11 and the lens unit of the second imaging system 12 are inserted and fixed into the two rectangular insertion hole portions formed through the connecting member 400.

[0036] Next, the configuration of a SPAD type imaging element used as the first imaging element 103 and the circuit configuration of a pixel will be described with reference to Fig. 3. As shown in Fig. 3(a), the first imaging element 103 has a layered structure made up of a sensor substrate 201 and a circuit substrate 202. Note that, although the first imaging element 103 has a layered structure in this embodiment, it is not limited to the layered structure and may have a single layer structure as long as it has the same function.

[0037] A pixel array is formed on the sensor substrate 201, in which a plurality of pixels 203 are arranged in rows and columns. Each pixel 203 is provided with a color filter array in a Bayer arrangement of, for example, R (red), G (green), and B (blue).

[0038] A pixel control unit 204, a signal processing circuit 205, and a substrate memory 206 are formed on the circuit substrate 202. The pixel control unit 204 is electrically connected to each pixel 203 of the sensor substrate 201 by a bump or the like, and outputs a control signal for driving the pixel 203 and receives a pulse waveform that is a buffer output from the pixel 203.

[0039] The pixel control unit 204 is provided with a counter for each corresponding pixel 203 that determines the presence or absence of photons by comparing the output of the pixel 203 with a preset threshold Vth and counts the number of pulse waveforms that change beyond the threshold.

[0040] The count value calculated by the pixel control unit 204 is output to the outside of the first imaging element 103 by the signal processing circuit 205. In addition, the board memory 206 is a volatile memory such as a DRAM, and is used for the purpose of temporarily storing data when the signal from the pixel control unit 204 is processed by the signal processing circuit 205.

[0041] Next, a description will be given of the configuration of the pixel 203. Fig. 3(b) is an equivalent circuit diagram of the pixel 203 formed on the sensor substrate 201. The pixel 203 is composed of a quench resistor 301, an avalanche photodiode (APD) 302 (light receiving element), and a buffer 303.

[0042] A reverse bias voltage of potential HVDD is applied to the APD 302 via a quench resistor 301. The potential HVDD at this time is set so that the reverse bias voltage is equal to or higher than the breakdown voltage in order to drive the APD 302 in the Geiger mode. The output of the buffer 303 is input to a counter 304 in the pixel control unit 204.

[0043] Here, the operation of the pixel 203 when a photon is incident thereon will be briefly described with reference to FIG.

[0044] 4(a) shows the current-voltage characteristics of the APD 302. In this embodiment, a potential HVDD is applied to the cathode of the APD 302 via a quench resistor 301 to create a reverse bias voltage exceeding the breakdown voltage, and the APD 302 enters the Geiger mode. When a photon is incident on the APD 302, a large current (photocurrent) flows in the APD 302 due to avalanche amplification (operation A).

[0045] At the same time as this current flows, the reverse bias voltage is dropped by the quench resistor 301, the reverse bias voltage applied to the APD 302 becomes less than the breakdown voltage, and the avalanche amplification stops (operation B). When the avalanche amplification stops, the cathode of the APD 302 is charged again by the potential HVDD, and the device returns to the Geiger mode (operation C).

[0046] The voltage change at the buffer input terminal due to operations A to C is pulse-shaped by the buffer 303 and measured by the counter 304. By repeating this process, it is possible to measure the number of photons incident on the APD 302. When viewing a high-luminance subject or driving a moving image, avalanche amplification occurs repeatedly, and power consumption due to the large current caused by avalanche amplification flowing through the quench resistor becomes an issue.

[0047] FIG. 4(b) is a schematic diagram showing the relationship between the pulse waveform of the output voltage due to avalanche amplification output from the APD 302 when a photon is incident, and the decision threshold Vth for deciding the incidence of a photon, with the horizontal axis being time. FIG. 4(b) shows the case where a potential HVDD is supplied to the APD 302, which can apply a reverse bias voltage that sufficiently exceeds the breakdown voltage. At this time, avalanche amplification occurs for each of the photons A (time t1), B (time t2), and C (time t3) incident on the APD 302, to the extent that a pulse waveform that changes beyond the decision threshold Vth of the counter is output, and each pulse is time-resolved. Therefore, in the case of FIG. 4(b), the number of photon events can be counted.

[0048] Next, the configuration of a CMOS type imaging element used in the second imaging element 123 and the circuit configuration of the pixels will be described with reference to FIG.

[0049] Fig. 5(a) is a block diagram showing the configuration of a CMOS imaging element equipped with a column AD conversion unit. As shown in Fig. 5(a), the CMOS imaging element, which is the second imaging element 123, has a plurality of pixels 501 arranged in a matrix of M rows and N columns, which convert light into electric charges and accumulate them on the light receiving surface. Each of the plurality of pixels 501 is provided with a color filter of any one of R (red), Gr, Gb (green), and B (blue) in a Bayer array.

[0050] Further, a column signal line 502 for transmitting an image signal corresponding to the amount of charge accumulated in each pixel 501 is formed for each column, and a column circuit section 503 is connected in series to each column signal line 502 .

[0051] The column circuit section 503 is composed of an amplifier, a CDS (Correlated Double Sampling) circuit, and an analog-to-digital (AD) conversion section, all of which are not shown. Each AD conversion section converts an analog image signal into a digital signal and outputs the digital signal. The digital signals are sequentially output by the column scanning circuit 504 through a horizontal signal line 505 and input to the second imaging signal processing section 124.

[0052] Furthermore, the row scanning circuit 506 receives a timing control signal from a timing control circuit 507 which operates according to a control signal from the control unit 13, and scans a transfer signal line 508, a reset signal line 509, and a row selection signal line 510. Then, the pixel signals output from each pixel 501 are read out to a column signal line 502 for each column on a row-by-row basis.

[0053] Next, the configuration of the pixel 501 will be described. Fig. 5(b) is an equivalent circuit diagram of the pixel 501. The pixel 501 is provided with a photodiode PD51, a transfer transistor M52, an amplification transistor M53, a selection transistor M54, and a reset transistor M55. Note that, here, each transistor is a switch element constituted by an n-channel MOSFET.

[0054] Further, a transfer signal line 508, a reset signal line 509, and a row selection signal line 510 are connected to the gates of the transfer transistor M52, the reset transistor M55, and the selection transistor M54, respectively. These signal lines extend in the horizontal direction and drive the pixels 501 in the same row simultaneously, thereby making it possible to control the operation of a line sequential operation type rolling shutter or an all-row simultaneous operation type global shutter. Furthermore, a column signal line 502 is connected to the source of the selection transistor M54, and one end of the column signal line 502 is grounded via a constant current source 56.

[0055] The photodiode PD51 performs photoelectric conversion and accumulates the generated charge, with its P side grounded and its N side connected to the source of the transfer transistor M52. When the transfer transistor M52 is turned ON, the charge accumulated in the photodiode PD51 is transferred to a floating diffusion section (FD) 57, but since the FD57 has a parasitic capacitance C58, the charge accumulates in this section.

[0056] The drain of the amplifier transistor M53 is connected to the power supply voltage Vdd, and the gate is connected to the FD 57. The amplifier transistor M53 converts the voltage of the FD 57 into an electric signal.

[0057] The selection transistor M54 is for selecting pixels from which signals are to be read out on a row-by-row basis, and has its drain connected to the source of the amplification transistor M53 and its source connected to the column signal line 502. When the selection transistor M54 is turned ON, the amplification transistor M53 and the constant current source 56 form a source follower, so that a voltage corresponding to the voltage of the FD57 is output to the column signal line 502.

[0058] The drain of the reset transistor M55 is connected to the power supply voltage Vdd, and the source is connected to the FD 57. The reset transistor M55 resets the photodiode PD51 to the power supply voltage Vdd via the FD 57 and the transfer transistor M52.

[0059] Next, the operational flow of still image shooting and moving image shooting in the imaging device 1 of this embodiment will be described with reference to FIG.

[0060] Compared to the pixel 501 of the CMOS type second imaging element 123 shown in FIG. 5, the pixel 203 of the SPAD type first imaging element 103 shown in FIG. 3 does not include a reset transistor M55 or an amplification transistor M53.

[0061] Therefore, the first imaging element 103 does not generate kTC noise or RTS noise caused by each transistor, and has a better S / N ratio than the CMOS type second imaging element 123, making it excellent for capturing still images where random noise has a large impact on image quality.

[0062] On the other hand, when taking videos or continuous shots, which involves repeatedly shooting many images, avalanche amplification causes a large current, resulting in a large amount of power consumption. This is likely to result in a reduction in the number of images that can be taken and a shortened operating time when the imaging device is battery-powered.

[0063] Therefore, in the first embodiment, when still image shooting using multiple pixels is selected, shooting is performed using a SPAD type first imaging element 103, and when video shooting, which consumes a large amount of power per unit time, is selected, shooting is performed using a CMOS type second imaging element 123.

[0064] When the imaging device 1 is powered on and imaging is started, first in S101, the control unit 13 determines whether the still image mode or the moving image mode has been selected by a user operation on the operation unit 14. If the still image mode has been selected, the process proceeds to S102, and if the moving image mode has been selected, the process proceeds to S103.

[0065] In S102, since the still image mode is selected, the first imaging system 11 is operated for still image shooting, and the still image shooting operation using the first imaging element 103 of the SPAD type is started.

[0066] On the other hand, in S103, since the moving image mode is selected, the second imaging system 12 is operated for moving image shooting, and the moving image shooting operation using the second CMOS image sensor 123 is started.

[0067] In this way, by using the first imaging system 11 and the second imaging system 12 depending on whether the imaging mode is a still image mode or a moving image mode, the total power consumption of the imaging device 1 is reduced and both still image and moving image capture is achieved.

[0068] Next, in S104, the control unit 13 determines whether or not to perform a recording operation in a mode selected by a user operation on the operation unit 14 or the like.

[0069] If recording is to be performed, the process transitions to S105, in which the image signal obtained by shooting in the mode determined in S101 is recorded by the image recording unit 16, and the process transitions to S106. On the other hand, if recording is not to be performed, the process transitions directly to S106.

[0070] In S106, the control unit 13 determines whether or not to end the image capture based on a user operation on the operation unit 14, and if the image capture is to be continued, the process returns to S101, and if the image capture is to be ended, the process ends.

[0071] As described above, according to the first embodiment, in an imaging device equipped with a SPAD type imaging element and a CMOS type imaging element, it is possible to achieve both image quality and power consumption by switching between still image shooting and video shooting.

[0072] In this embodiment, the first imaging element 103 (SPAD type imaging element) or the second imaging element 123 (CMOS type imaging element) is controlled to be used by selecting the still image mode or the video mode, but the present invention is not limited to this. For example, the imaging element 103 may be switched according to the shooting mode, such as the difference in video resolution or single shot and continuous shot, depending on the power consumption and the upper limit of the temperature rise allowed for the imaging device. That is, when a shooting mode for shooting high-resolution video is set, the first imaging element 103 (SPAD type imaging element) is controlled to be used, and when a shooting mode for shooting low-resolution video is set, the second imaging element 123 (CMOS type imaging element) is controlled to be used. Alternatively, when a single shot shooting mode is set, the first imaging element 103 (SPAD type imaging element) is controlled to be used, and when a continuous shot shooting mode is set, the second imaging element 123 (CMOS type imaging element) is controlled to be used.

[0073] Also, the switching may be performed according to the shutter speed during still image shooting and the frame rate during video shooting. That is, when still image shooting is performed at a shutter speed faster than a predetermined shutter speed, the first imaging element 103 (SPAD type imaging element) is controlled to be used, and when still image shooting is performed at a shutter speed slower than the predetermined shutter speed, the second imaging element 123 (CMOS type imaging element) is controlled to be used. Furthermore, when the frame rate during video shooting is higher than the predetermined frame rate, the first imaging element 103 (SPAD type imaging element) is controlled to be used, and when the frame rate during video shooting is lower than the predetermined frame rate, the second imaging element 123 (CMOS type imaging element) is controlled to be used.

[0074] <Modification of the first embodiment> Fig. 7 is a diagram showing an example of the appearance of an imaging device in a modified example of the first embodiment. The imaging device 1 shown in Fig. 1 has a configuration in which light from a subject is incident on two imaging elements, respectively, using two optical systems. In contrast, the imaging device 7 shown in Fig. 7 receives light from a subject through one optical system, and guides the light to two imaging elements by reflecting or splitting the light using a light guide means such as a mirror within the imaging device 7.

[0075] Fig. 8 is a block diagram showing the configuration of the imaging device 7 shown in Fig. 7. In Fig. 8, the same components as those in Fig. 1 are given the same reference numerals, and the description thereof will be omitted.

[0076] 8, the imaging device 7 in this modification includes an optical barrel 81, a shutter mechanism section 102, a first imaging element 103, a second imaging element 123, a control section 13, an operation section 14, an image display section 15, an image recording section 16, and a mirror unit 82. Note that in this modification, the control section 13 will be described as having various functions of signal processing upon receiving image signals from the first imaging element 103 and the second imaging element 123, and compressing and expanding imaging data.

[0077] The optical barrel 81 includes a lens for collecting light from a subject onto the first imaging element 103 or the second imaging element 123 configured in the imaging device 7, and an optical mechanism section 811. The optical mechanism section 811 is driven based on a control signal from the control section 13, and performs focus adjustment, changes in the optical imaging magnification, adjustment of the amount of incident light, etc.

[0078] Mirror unit 82 is composed of a mirror section 821 and a mirror driver 822 , and serves to guide the light incident from optical barrel 81 to first imaging element 103 or second imaging element 123 .

[0079] The mirror driving unit 822 drives the mirror unit 821 by an actuator or the like in accordance with a control signal from the control unit 13. That is, it drives the mirror unit 821 to be located at either a first position (mirror down) on the optical axis of the lens shown in Fig. 8 or a second position (mirror up) where the mirror unit 821 is flipped up and retracted from the optical axis.

[0080] When mirror section 821 is in the first position, it reflects light from optical barrel 81 and causes the light to be incident on second imaging element 123. On the other hand, when mirror section 821 is in the second position, the light from optical barrel 81 is incident on first imaging element 103 as it is.

[0081] At this time, the configuration position of mirror section 821 is at an optically equivalent position with respect to the imaging planes of first imaging element 103 and second imaging element 123. In other words, it can be said that first imaging element 103 arranged on the first imaging plane and second imaging element 123 arranged on the second imaging plane are each at an image formation plane optically conjugate with the subject via optical barrel 81.

[0082] Fig. 9 is a flow chart for explaining the operation of the imaging device 7 having the configuration shown in Fig. 8. Note that the same processes as those in Fig. 6 are given the same step numbers, and the explanation will be omitted as appropriate.

[0083] Imaging is started, and in S101, if the still image mode is selected, the process proceeds to S901, and if the moving image mode is selected, the process proceeds to S903.

[0084] When the still image mode is selected, the mirror portion 821 is driven to the second position (mirror up position) in S901, and the light from the optical barrel 81 is guided to the first imaging element 103.

[0085] In S901, the mirror unit 821 is driven to the second position, and then the process proceeds to S902 where a still image capturing operation using the first SPAD image capturing element 103 is started for still image capturing, and then the process proceeds to S104.

[0086] On the other hand, if the moving image mode is selected, the mirror portion 821 is driven to the first position (mirror down position) in S903, and the light from the optical barrel 81 is guided to the second imaging element 123.

[0087] In S903, the mirror portion 821 is driven to the first position, and then the process proceeds to S904 to start a moving image capturing operation using the second CMOS image capturing element 123 for capturing a moving image, and then the process proceeds to S104.

[0088] The process from S104 onwards is similar to the process described above in FIG. 6, and therefore the description will be omitted.

[0089] As described above, according to the modification of the first embodiment, in addition to providing the same effects as the first embodiment, it can be configured with a single optical system.

[0090] It should be noted that mirror section 821 does not need to be a total reflection mirror, and may be configured, for example, to employ a half mirror to split the light beam from optical barrel 81 and cause the light beam to be incident simultaneously on first imaging element 103 and second imaging element 123. In that case, the configuration does not require mirror operation, and the operation described in the first embodiment with reference to FIG.

[0091] <Second embodiment> Next, a second embodiment of the present invention will be described. In the second embodiment of the present invention, the imaging device 1 described in the first embodiment with reference to Figs. 1 to 5 or the imaging device 7 described in the modified example of the first embodiment with reference to Figs. 7 and 8 can be used, so a description thereof will be omitted here.

[0092] When the imaging device 7 is used, it can be applied to the second embodiment by using a half mirror instead of a total reflection mirror. Hereinafter, the process in the second embodiment will be described with reference to Figs. 10 to 12.

[0093] 10 and 11 are diagrams for explaining a case where a count error occurs in a SPAD type imaging element.

[0094] Fig. 10(a) is a schematic diagram showing the relationship between the pulse waveform of the output voltage due to avalanche amplification when a photon is incident and the decision threshold Vth, with the horizontal axis representing time. Unlike the behavior during normal operation described in Fig. 4(b), after a voltage that fluctuates beyond the decision threshold Vth is input due to photon D (time t4), photon E (time t5) is incident before the avalanche amplification operation in operation B described in Fig. 4(a) stops. At this time, photon E is incident at time t5 before the waveform fluctuation due to avalanche amplification that occurred at time t4 exceeds the decision threshold Vth, so the count operation for photon E cannot be performed.

[0095] Furthermore, when photon F (at time t6) is incident, the same state occurs as between times t4 and t5, so photon F is also not counted. In this way, when the luminance of the subject is high, photons are incident continuously before exceeding the judgment threshold Vth, so the count value becomes smaller than the number of photons that actually incident, resulting in a count error (count saturation).

[0096] Note that, between times t6 and t7, no photons are incident and the voltage exceeds the determination threshold Vth once, so the pulse waveform of the voltage for the subsequently incident photon G (time t7) is counted.

[0097] Fig. 10(b) is a diagram showing the relationship between illuminance and count value in a SPAD type imaging element. As the illuminance increases, the number of photons increases, and the count value counted by the SPAD type imaging element also increases proportionally, but when the illuminance exceeds M, the state from time t4 to time t6 in Fig. 10(a) occurs, and a count error (count saturation) occurs. If the illuminance increases further, the number of simultaneously incident photons increases further, and the count error state continues, so that the actual count value (solid line) is inversely proportional to the ideal count value (dashed line) at the illuminance N boundary.

[0098] At illuminance levels M and above, when shooting a scene that includes a high-brightness subject, as shown in Figure 11(a), the illuminance of the high-brightness subject will result in an image with darkening, as shown in the gradation area in Figure 11(b).

[0099] Therefore, in the second embodiment, depending on the brightness of the subject, image portions in which count errors occur for high-brightness subjects in an image obtained by the first SPAD-type imaging element 103 are interpolated with an image obtained by the second CMOS-type imaging element 123.

[0100] The imaging operation in the second embodiment will be described with reference to the flowchart in FIG.

[0101] When the power supply to the imaging device is turned on and imaging is started, first in S201, the control unit 13 determines whether or not to perform imaging and recording in response to an instruction to perform imaging and recording by a user's operation on the operation unit 14.

[0102] If it is determined in S201 that shooting and recording are to be performed, then in S203, shooting is performed using the first imaging element 103 and, in parallel, shooting is performed using the second imaging element 123 to obtain images from each. On the other hand, if shooting and recording are not to be performed, the process proceeds to S209.

[0103] In this embodiment, the first imaging element 103 and the second imaging element 123 are used to capture images in parallel, but the images may be captured separately in time series.

[0104] Next, in S204, the second image capturing signal processing unit 124 calculates a subject brightness value for each predetermined region based on the area distribution of image signal values ​​in the image acquired from the second image capturing element 123 in S203, the aperture setting value of the second optical mechanism unit 122, and the exposure time and sensitivity setting value of the second image capturing element 123.

[0105] Then, this subject luminance value is held as a correction judgment value in the control unit 13. Note that the above-mentioned regions may be appropriately set, for example, by detecting subjects using a known method and setting each subject as each region, or by dividing the image into a plurality of blocks of a predetermined size.

[0106] Next, in S205, the control unit 13 determines whether at least any of the correction determination values ​​for each region obtained in S204 is equal to or greater than the correction determination threshold at which a count error occurs in the first imaging element 103. If at least any of the correction determination values ​​is equal to or greater than the correction determination threshold, it is determined that correction is necessary, and if all of the correction determination values ​​are less than the correction determination threshold, it is determined that correction is not necessary. Note that the correction determination threshold is held in advance in the control unit 13 as a design value based on, for example, the characteristics of the first imaging element 103 described with reference to FIG. 10(b).

[0107] If it is determined in S205 that correction is not necessary, the process proceeds to S206, and if it is determined that correction is necessary, the process proceeds to S207.

[0108] In S206, since no correction is required, the image obtained by the first imaging element 103 in S202 is directly recorded by the image recorder 16, and the process proceeds to S209.

[0109] On the other hand, in S207, it is determined that the image obtained by the first imaging element 103 in S205 contains a pixel signal having a count error. Therefore, in the image obtained by the first imaging element 103, the control unit 13 performs correction by replacing the image signal of the address where the correction determination value for each area obtained in S204 exceeds the correction determination threshold with the image signal of the area of ​​the second imaging element 123 corresponding to that address. After the correction, the process proceeds to S209.

[0110] The difference in signal level between the images obtained from the first imaging element 103 and the second imaging element 123 may be converted by, for example, acquiring in advance the difference in signal level between the second imaging element 123 and the first imaging element 103 obtained for each luminance as a table or function. Also, when the area where a count error occurs in the image obtained by the first imaging element 103 is narrow, the count error may be corrected by performing interpolation from surrounding pixel signals where no count error occurs, or by replacing the signal value of the pixel area where the count error occurs with a fixed value. Various other correction methods are possible, and the present invention is not limited by the correction method.

[0111] In S209, the control unit 13 determines whether or not to end the shooting. If it is to be ended, the control unit 13 transitions to a standby state, and if it is not to be ended, the control unit 13 returns to the process of S201 to continue the operation.

[0112] In the above example, the subject luminance value for each region is calculated as the correction judgment value and compared with the correction judgment threshold, but the present invention is not limited to this. For example, the signal value difference or signal value ratio of the same address region of the images obtained by the first imaging element 103 and the second imaging element 123 may be used as the correction judgment value. Also, the maximum value of the subject luminance value obtained for each region may be compared with the correction judgment threshold as the correction judgment value to make a judgment.

[0113] Also, the maximum luminance value in the image obtained from the second imaging element 123 may be compared as a correction judgment value with a correction judgment threshold, and if it is judged that correction is necessary, the subject luminance value for each region may be judged. That is, the luminance value of the image obtained from the second imaging element 123 may be used to judge the presence or absence of a count error in the image obtained from the first imaging element 103 and correct it, and various other judgment methods, correction methods, and procedures may also be considered.

[0114] As described above, according to the second embodiment, count errors caused by high-brightness subjects that occur when using a SPAD type imaging element can be detected and the image can be corrected, thereby providing images with good image quality.

[0115] <Third embodiment> Next, a third embodiment of the present invention will be described. In the third embodiment of the present invention, the imaging device 1 described in the first embodiment with reference to Figs. 1 to 5 or the imaging device 7 described in the modified example of the first embodiment with reference to Figs. 7 and 8 can be used, so a description thereof will be omitted here.

[0116] When the imaging device 7 is used, it is possible to apply the third embodiment by using a half mirror instead of a total reflection mirror. Hereinafter, the process in the third embodiment will be described with reference to Figs. 13 and 14.

[0117] In the third embodiment, an operation of performing dynamic range expansion processing (hereinafter referred to as "HDR processing") using image signals obtained from the first SPAD type imaging element 103 and the second CMOS type imaging element 123 will be described. Note that in the present embodiment, the HDR processing will be described as being performed within the control unit 13.

[0118] Fig. 13 is a diagram showing the relationship between the amount of incident light and the amount of signal due to HDR processing. In general, HDR processing is a process for expanding the dynamic range of a signal by using images captured under low-exposure and high-exposure imaging conditions.

[0119] As described in the description of the background art and the second embodiment, the SPAD type imaging element improves the S / N ratio by not generating RTS noise, but is prone to count errors on the high luminance side.

[0120] Therefore, in the HDR processing of the third embodiment, the dynamic range is expanded by using image signals from a CMOS image sensor in a signal range where the amount of incident light is large, and by using image signals from a SPAD image sensor which provides a signal with a high S / N ratio on the low-luminance side where the amount of incident light is small.

[0121] In the first SPAD imaging element 103, the signal level Q3 at which count errors begin to occur is reached when the amount of incident light reaches P2, and in the second CMOS imaging element 123, the saturation signal level Q2 in the CMOS imaging element is reached when the amount of incident light reaches P3. On the other hand, if the signal level obtained by receiving light is equal to or lower than Q1, it corresponds to the noise level, and therefore the pixel signal cannot be used.

[0122] Therefore, the dynamic range of the first imaging element 103 is a range in which the amount of incident light ranges from P0 to P2, and the dynamic range of the second imaging element 123 is a range in which the amount of incident light ranges from P1 to P3.

[0123] Here, it is assumed that the ratio of the signal amount between the first imaging element 103 and the second imaging element 123 is 3:1. In this case, the control unit 13 calculates a pixel signal HDL_A after HDR processing for the signal amount within the range of the incident light amount A (light amount P0 to P1, low level) for a certain pixel in the imaging screen, using the following formula (1). Pixel signal HDL_A = 1st image sensor pixel signal × 1 + 2nd image sensor pixel signal × 0 … (1) Furthermore, for a signal amount within the range of incident light amount B (light amount P1 to P2, medium level) for a certain pixel in the imaging screen, the control unit 13 calculates a pixel signal HDL_B after HDR processing using the following formula (2). Pixel signal HDL_B = first image sensor pixel signal × (1-α) + 2nd image sensor signal × α × 3 … (2) Furthermore, for a signal amount within the range of the incident light amount C (light amount P2 to P3, high level) for a certain pixel in the imaging screen, the control unit 13 calculates a pixel signal HDL_C after HDR processing using the following formula (3). Pixel signal HDL_C = 1st image sensor pixel signal × 0 + 2nd image sensor pixel signal × 3 … (3)

[0124] As described above, the control unit 13 classifies the signal amount of each pixel in the imaging screen into three levels, for example, low level, medium level, and high level. Then, for pixel signals corresponding to an incident light amount with a low signal amount, a pixel signal after HDR processing is calculated by equation (1) using only the pixel signal of the first imaging element 103.

[0125] For pixel signals corresponding to a medium level of incident light intensity, the control unit 13 determines a pixel signal after HDR processing using equation (2) in which the pixel signal of the first imaging element 103 and the pixel signal of the second imaging element 123 are combined at a ratio of (1-α):α, where α (α is 0 to 1) represents the combination ratio. For pixel signals corresponding to a high level of incident light intensity, the control unit 13 determines a pixel signal after HDR processing using equation (3) that uses only the pixel signal of the second imaging element 123.

[0126] This makes it possible to generate a high dynamic range image in which the signal amount is expanded from Q1 to Q4 as shown in Fig. 13. The division of the signal amount level into low, medium and high levels is determined in advance according to the characteristics of the first imaging element 103 and the second imaging element 123.

[0127] Next, HDR processing in the imaging device according to the third embodiment will be described with reference to the flowchart in FIG.

[0128] When the HDR mode is selected by a user operation and shooting is started, predetermined shooting conditions are set for the first imaging element 103 and the second imaging element 123 in S301.

[0129] Next, in S303, imaging is performed using the first imaging element 103, and in parallel, imaging is performed using the second imaging element 123 to obtain images. Note that, although this embodiment will be described as performing imaging and obtaining images in parallel using the first imaging element 103 and the second imaging element 123, they may be obtained separately in chronological order.

[0130] Next, in S304, the control unit 13 performs the above-mentioned HDR processing using the images obtained by the first imaging element 103 and the second imaging element 123 in S303.

[0131] Next, in S305, it is determined whether or not to end the HDR processing, and if it is to be ended, the process transitions to a standby state. On the other hand, if shooting is to be continued, the process returns to S301 and shooting is continued.

[0132] In this embodiment, an imaging device having a 2-size imaging element configuration in which a SPAD type imaging element and a CMOS type imaging element are separately configured is described, but the present invention is not limited to this. For example, a 1-size imaging element configuration may be configured with SPAD type imaging pixels and CMOS type imaging pixels alternately, and HDR processing may be performed using image signals obtained from rows of SPAD type imaging pixels and rows of CMOS type imaging pixels that are physically adjacent to each other. In this case, the SPAD type imaging pixels and the CMOS type imaging pixels may be configured alternately for one or more rows, or for one or more columns, or may be configured in a checkerboard pattern.

[0133] In the above example, the pixel signal after HDR processing is acquired using one of the formulas (1) to (3) according to the amount of incident light, but the present invention is not limited to this, and it may be determined which of the formulas (1) to (3) to use according to the amount of signal. In this case, for example, a low level, a medium level, or a high level is determined based on the amount of signal obtained from the first imaging element 103 and the amount of signal obtained from the second imaging element 123, and one of the formulas (1) to (3) is used based on the determination result.

[0134] As described above, according to the third embodiment, by performing HDR processing as described above using a SPAD type image sensor and a CMOS type image sensor, it is possible to obtain an image with a dynamic range suited to the luminance of a subject.

[0135] <Fourth embodiment> Next, a fourth embodiment of the present invention will be described. In the fourth embodiment of the present invention, the imaging device 1 described in the first embodiment with reference to Figs. 1 to 5, or the imaging device 7 described in the modified example of the first embodiment with reference to Figs. 7 and 8 can be used, so a description thereof will be omitted here. Hereinafter, the process in the fourth embodiment will be described with reference to Fig. 15.

[0136] In the fourth embodiment, imaging is switched between using a SPAD type imaging element or a CMOS type imaging element depending on the sensitivity setting in the imaging conditions. The configuration of the pixel 203 of the first SPAD type imaging element 103 described in Fig. 3 is a configuration that does not require a reset transistor M55 or an amplification transistor M53, compared to the configuration of the pixel 501 of the second CMOS type imaging element 123 described in Fig. 5.

[0137] Therefore, since kTC noise, RTS noise, etc. caused by these configurations do not occur, the S / N ratio is superior to that of the CMOS type. However, as described in the second embodiment with reference to Fig. 10, count errors occur under conditions where a large number of photons are incident per unit time.

[0138] Therefore, in the imaging device of the fourth embodiment, when shooting under exposure conditions where a large number of photons are incident on the pixels, imaging is performed using a CMOS type imaging element, and when shooting under exposure conditions such as low illuminance, where a large amount of noise is likely to occur, imaging is performed using a SPAD type imaging element.

[0139] First, when the power is turned on to the imaging device and imaging starts, in S401, imaging operation is performed by the second imaging element 123 to determine the exposure value. In S401, since the brightness of the subject is unknown immediately after the start of imaging, the second imaging element 123, which is a CMOS type that does not cause problems due to high brightness, is used to acquire the image.

[0140] Next, in S402, a subject luminance value Ex is obtained from the image obtained in S401. In this embodiment, the subject luminance value Ex is calculated based on the image signal obtained by the second image sensor 123, the exposure time and sensitivity setting value set in the second image sensor 123, and the setting value of the optical aperture in the second optical mechanism unit 122.

[0141] In S402, if processing is based on the image signal from the second imaging element 123, the processing is performed by the second imaging signal processing unit 124, but if transitioning from S408 described later and processing is based on the image signal from the first imaging element 103, calculation is performed by the first imaging signal processing unit 115.

[0142] Next, in S403, the control unit 13 determines whether the subject luminance value Ex calculated in S402 is equal to or greater than a predetermined threshold value Eth. Note that this threshold value Eth is set to a luminance at which the SPAD-type first imaging element 103 causes a counting error or a luminance at which the S / N ratio of the CMOS-type second imaging element 123 exceeds an allowable value.

[0143] If the subject luminance value Ex is less than the threshold value Eth in S403, the process proceeds to S404, and shooting is performed using the SPAD-type first imaging element 103. On the other hand, if the subject luminance value Ex is equal to or greater than the threshold value Eth in S403, the control unit 13 proceeds to S405, and shooting is performed using the CMOS-type second imaging element 123.

[0144] Next, in S406, the control unit 13 determines whether or not to perform a recording operation based on a user operation via the operation unit 14. If recording is to be performed, the process proceeds to S407, in which the image signal obtained by shooting in S404 or S405 is recorded by the image recording unit 16, and the process proceeds to S408. On the other hand, if recording is not to be performed, the process proceeds directly to S408.

[0145] In S408, the control unit 13 determines whether or not to end the image capture based on a user operation on the operation unit 14, and if the image capture is to be continued, the process returns to S402, and if the image capture is to be ended, the process ends.

[0146] When the process transitions from S408 to S402, the subject luminance value Ex is obtained from the image signal recorded in S407. Therefore, when the image signal from the first image sensor 103 in S404 is used, the subject luminance value Ex is obtained by the first image signal processing unit 115 as described above.

[0147] In the above example, the subject brightness value Ex is calculated in S402 and the imaging operation is switched in S403 based on the value, but the imaging operation may be switched according to imaging conditions such as the sensitivity set in the imaging element. In this case, the imaging condition setting may be determined by the control unit 13 based on a photometric value calculated using an image signal or an external measurement element such as a photometric sensor (not shown), or may be determined by a user's operation. Then, if the sensitivity is higher than a predetermined sensitivity, the process may proceed to S404, and if the sensitivity is equal to or lower than the sensitivity threshold, the process may proceed to S405.

[0148] Furthermore, the imaging element to be used may be switched depending on the aperture value set according to the depth of field, shutter speed, etc., the presence or absence of an ND filter for adjusting the amount of incident light, or the density of the ND filter to be inserted, etc. That is, when an aperture value narrower than a predetermined aperture value is set, the first imaging element 103 (SPAD type imaging element) is controlled to be used, and when an aperture value more open than the predetermined aperture value is set, the second imaging element 123 (CMOS type imaging element) is controlled to be used.

[0149] Also, when an ND filter is inserted on the optical path, the first imaging element 103 (SPAD type imaging element) is controlled to be used, and when an ND filter is not inserted on the optical path, the second imaging element 123 (CMOS type imaging element) is controlled to be used. Alternatively, when an ND filter with a density higher than a predetermined density is inserted on the optical path, the first imaging element 103 (SPAD type imaging element) is controlled to be used, and when an ND filter with a density equal to or lower than a predetermined density is inserted on the optical path, the second imaging element 123 (CMOS type imaging element) is controlled to be used.

[0150] As described above, according to the fourth embodiment, high-quality images can be provided by avoiding high-luminance dips due to counting errors in a SPAD-type imaging element and image quality degradation due to a decrease in S / N in a CMOS-type imaging element.

[0151] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0152] <Other embodiments> The present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device.

[0153] The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0154] 11: first imaging system, 103: first imaging element, 115: first imaging signal processing unit, 12: second imaging system, 123: second imaging element, 124: second imaging signal processing unit, 13: control unit, 14: operation unit

Claims

1. a first image sensor having a plurality of pixels that counts the number of incident photons and outputs the counted value as a first color image signal; a second image sensor having a plurality of pixels for outputting an electrical signal corresponding to an amount of charge obtained by photoelectrically converting incident light as a second color image signal; An acquisition means for acquiring a luminance value; a generating means for generating a color image based on the first color image signal or the second color image signal; a control means for controlling the generating means to generate a color image based on the second color image signal when the luminance value is equal to or greater than a predetermined threshold value, and to generate a color image based on the first color image signal when the luminance value is less than the threshold value; An imaging device comprising:

2. A first image sensor having a plurality of pixels that counts the number of incident photons and outputs the counted value as a first color image signal; a second image sensor having a plurality of pixels for outputting an electrical signal corresponding to an amount of charge obtained by photoelectrically converting incident light as a second color image signal; An acquisition means for acquiring a luminance value; a generating means for generating a color image based on the first color image signal and the second color image signal; a control means for controlling the generation means to generate a color image using the first color image signal when the luminance value is less than a predetermined first threshold, to generate a color image by combining the first color image signal and the second color image signal when the luminance value is equal to or greater than the first threshold and less than a predetermined second threshold that is greater than the first threshold, and to generate a color image using the second color image signal when the luminance value is equal to or greater than the predetermined second threshold; An imaging device comprising:

3. Further, a first imaging optical system that guides light from a subject to the first imaging element; 2. The imaging apparatus according to claim 1, further comprising: a second imaging optical system different from the first imaging optical system, which guides light from the subject to the second imaging element.

4. The imaging device described in Claim 3, characterized in that the control means controls the first imaging optical system to guide light from the subject to the first imaging element when the generation means selects the first color image signal to generate a color image, and controls the second imaging optical system to guide light from the subject to the second imaging element when the generation means selects the second color image signal to generate a color image.

5. Further, a first imaging optical system that guides light from a subject to the first imaging element; 3. The imaging apparatus according to claim 2, further comprising: a second imaging optical system different from the first imaging optical system, which guides light from the subject to the second imaging element.

6. The imaging device described in claim 1 or 2, further characterized in having a light guiding means for splitting light incident through the imaging optical system and causing it to be incident on the first imaging element and the second imaging element.

Citation Information

Patent Citations

  • Imaging apparatus

    JP2006352804A

  • Image pickup apparatus and driving method of imaging device

    JP2007006071A

  • Solid-state imaging apparatus

    JP2008193163A

  • CMOS imaging device with single-photon counting function

    JP2012513694A

  • Photodetector

    JP2014081253A