Imaging device, control method for imaging device, program and storage medium

Abstract

To suppress the decrease in visibility of moving subjects. [Solution] The imaging device includes an image sensor having a counter circuit that counts pulses emitted from a sensor unit that emits pulses in response to incident photons, a determination circuit that determines whether the count value is above a predetermined threshold at each of a set of multiple determination times, and an output circuit that outputs the count value and the determination result from the determination circuit. It also includes identification means for identifying pixels or regions in two frames output from the image sensor where there is a difference between the determination result of the determination circuit and the determination result of the second determination corresponding to each of the two frames, and control means for changing the imaging settings of the image sensor to reduce the number of pixels or regions identified.

Description

Technical Field

[0001] The present invention relates to an imaging device, a control method for the imaging device, a program, and a storage medium.

Background Art

[0002] In recent years, an imaging device has been proposed that performs photoelectric conversion by digitally counting the number of photons arriving at an avalanche photodiode (hereinafter, APD) and outputting the count value from each pixel. Patent Document 1 discloses a technique for suppressing the power consumption of an imaging device even when photons are incident on the APD at a high frequency by resetting a reset unit with a clock pulse at a constant period. Further, when the count values corresponding to each exposure time reach a predetermined threshold value at a plurality of exposure times shorter than the maximum exposure time, the counting of the number of photons is stopped, and an estimated value of the count is calculated and substituted to reduce the power consumption. Here, the estimated value of the count is a count value obtained by estimating the number of photons assuming that the APD is exposed for the length of the maximum exposure time, based on the count value at an exposure time shorter than the maximum exposure time.

[0003] By using the photoelectric conversion technology disclosed in Patent Document 1 as described above, it is possible to detect even a small amount of light in a dark place and capture an image with a very wide dynamic range.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the imaging device disclosed in Patent Document 1, if there is a large difference in brightness between the moving subject and the area surrounding the subject, and the exposure time differs, the amount of blur of the moving subject differs between frames, and the visibility of the moving subject decreases.

[0006] Therefore, the problem that this invention aims to solve is to suppress the decrease in visibility of moving subjects. [Means for solving the problem]

[0007] To solve the above problems, an imaging device according to one aspect of the present invention includes an avalanche photodiode and a sensor unit that emits pulses in response to photons incident on the avalanche photodiode; a counter circuit that counts the pulses emitted from the sensor unit; a determination circuit that determines whether the count value of the counter circuit is greater than or equal to a predetermined threshold, wherein the determination circuit determines whether the count value is greater than or equal to the predetermined threshold at each of a set of multiple determination times; and an output circuit that outputs the count value and the determination result of the determination circuit; an acquisition means for acquiring the determination result of the determination circuit for two frames output from the imaging device, wherein the determination result is a first determination result and a second determination result corresponding to each of the two frames; a identification means for identifying a pixel or region in the imaging device that has a difference between the first determination result and the second determination result; and a control means for changing the imaging settings of the imaging device to reduce the pixel or region identified by the identification means. [Effects of the Invention]

[0008] According to the present invention, it is possible to suppress the decrease in visibility of moving subjects. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing the configuration of the imaging device. [Figure 2] This is a diagram showing the configuration of a photoelectric conversion element. [Figure 3] This is a diagram showing the configuration of the sensor board. [Figure 4] This is a diagram of the circuit board configuration. [Figure 5] This is a conceptual diagram used to explain the equivalent circuit of a signal processing circuit. [Figure 6] This is a conceptual diagram illustrating the relationship between the operation of the APD and its output signal. [Figure 7] This is a timing chart showing the operation of a signal processing circuit. [Figure 8] This figure shows the relationship between the exposure time for each pixel in the photoelectric conversion element and the count value of the counter circuit. [Figure 9] This figure shows an example of exposure time information output based on the exposure time when the count threshold is exceeded. [Figure 10] This figure shows the relationship between the exposure time for each pixel in the photoelectric conversion element and the count value of the counter circuit. [Figure 11] This is a functional configuration diagram of the imaging device. [Figure 12] This is a flowchart showing the processing of the imaging device. [Figure 13] This is a conceptual diagram to explain exposure time information. [Figure 14] This is a conceptual diagram illustrating the first exposure time information and the second exposure time information. [Figure 15] This figure shows an example of exposure time information output according to the exposure time when the count threshold of Embodiment 1 is exceeded. [Figure 16] This is a flowchart showing the processing of the imaging device. [Figure 17] This is a flowchart showing the processing of the imaging device. [Modes for carrying out the invention]

[0010] <First Embodiment> Hereinafter, a first embodiment of the present invention will be described. First, the configuration of the imaging device 100 in this embodiment will be described using FIG. 1. In FIG. 1, an optical lens 101, an imaging element 102, a CPU 103, a video output driving unit 104, a frame memory 106, a ROM 107, a RAM 108, an operation unit 109, and a display driving unit 110 are connected to an internal bus 112. Each unit connected to the internal bus 112 is enabled to exchange data with each other via the internal bus 112.

[0011] The optical lens 101 is an optical element composed of a lens and a motor for driving the lens. The optical lens 101 operates based on a control signal and can optically enlarge or reduce an image, or adjust parameters such as the focal length. Also, when it is desired to adjust the incident light amount, the aperture area of the diaphragm can be controlled to adjust the light amount to a desired brightness. The light transmitted through the lens is imaged by the imaging element 102.

[0012] The imaging element 102 has a color filter and a microlens (not shown), and uses a photoelectric conversion element described later to play a role of replacing an optical signal with an electrical signal.

[0013] The CPU 103 functions as a CPU (Central Processing Unit) for controlling each function of the imaging device 100. For example, it controls the imaging settings of the imaging device 100 and performs various image processes on the image output from the imaging element 102. Details will be described later.

[0014] The video output driving unit 104 outputs the image processed by the CPU 103 to the outside of the imaging device 100 via the video terminal 105.

[0015] Video terminal 105 is an interface that outputs video that can be viewed by the user. Typical interfaces include SDI (Serial Digital Interface) and HDMI (High Definition Multimedia Interface). Various other interfaces exist, such as DisplayPort (registered trademark), which allow for real-time video display on external monitors.

[0016] Frame memory 106, commonly known as RAM (Random Access Memory), is a component that temporarily stores video signals and allows them to be read when needed. Because video signals contain a vast amount of data, high speed and high capacity are required. In recent years, DDR4-SDRAM (Dual Data Rate 4 - Synchronous Dynamic RAM) is often used. Using this frame memory 106 enables various processing tasks. For example, it is an indispensable component for image processing, such as performing image processing using images from different time periods or extracting only the necessary areas.

[0017] ROM107 is a non-volatile element that stores programs for operating CPU103, as well as various adjustment parameters. Programs read from ROM107 are loaded into the volatile RAM108 and executed.

[0018] RAM108 is generally slower and has a lower capacity than frame memory 106.

[0019] The control unit 109 is primarily used by the user to change the settings menu of the imaging device 100. The control unit 109 may be operated via a physical interface such as buttons, analog keys, or switches, or via an external device such as a USB interface, or via a network using a web browser or the like.

[0020] The display driver unit 110 displays the image processed by the CPU 103 on the display device via the display unit 111.

[0021] The display unit 111 is a display device that can be viewed by the user. For example, it can display images processed by the CPU 103 or setting menus, allowing the user to check the operating status of the imaging device 100. In recent years, the display unit 111 has utilized small, low-power devices such as LCDs (Liquid Crystal Displays) and organic ELs (Electroluminescence) as display devices. Furthermore, it may also incorporate resistive or capacitive thin-film elements known as touch panels.

[0022] Next, the image sensor 102 in this embodiment will be described using Figures 2 to 10. Figure 2 is a diagram showing an example of the configuration of the image sensor 102 in this embodiment. In the following, the photoelectric conversion element 200 is constructed by stacking two substrates, a sensor substrate 21 and a circuit substrate 23, and electrically connecting them. An image sensor in which the photoelectric conversion section has a stacked structure will be used as an example for explanation.

[0023] The circuit board 23 includes a circuit region 24 for processing signals detected in the pixel region 22.

[0024] Figure 3 shows an example of the configuration of the sensor substrate 21. The pixel area 22 of the sensor substrate 21 includes multiple pixels 301 arranged in a two-dimensional manner across multiple rows and columns.

[0025] Each pixel 301 includes a photoelectric conversion unit 302 that incorporates an APD. The number of rows and columns of the pixel array forming the pixel region 22 is not particularly limited.

[0026] Figure 4 shows an example of the configuration of the circuit board 23. The circuit board 23 has a signal processing circuit 401 for processing the charge photoelectrically converted by the photoelectric conversion unit 302 in Figure 3, a readout circuit 402, a control unit 403, a horizontal scanning circuit unit 404, a signal line 405, and a vertical scanning circuit unit 406. The signal output from the photoelectric conversion unit 302 of the pixel is processed by the signal processing circuit 401.

[0027] The signal processing circuit 401 is equipped with a counter and memory, and the memory stores a digital value representing the number of photons counted.

[0028] The horizontal scanning circuit 404 inputs control pulses to the signal processing circuit 401 to sequentially select each column in order to read a signal from the memory of each pixel in which a digital value is held.

[0029] Signals are output to signal line 405 from the signal processing circuit 401 of the pixels selected by the vertical scanning circuit 406 for the column selected by the horizontal scanning circuit 404. The signals output to signal line 405 are then output to the outside of the photoelectric conversion element 200 via the output circuit 407.

[0030] As shown in Figures 3 and 4, multiple signal processing circuits 401 are arranged in the region that overlaps with the pixel region 22 in a plan view. The vertical scanning circuit section 406, horizontal scanning circuit section 404, readout circuit 402, output circuit 407, and control unit 403 are arranged so as to overlap between the edge of the sensor substrate 21 and the edge of the pixel region 22 in a plan view. These circuit sections and control units constitute a non-pixel region that does not include pixels 301, and the sensor substrate 21 has a pixel region 22 and a non-pixel region arranged around the pixel region 22. The vertical scanning circuit section 406, horizontal scanning circuit section 404, readout circuit 402, output circuit 407, and control unit 403 are arranged in the region that overlaps with the non-pixel region in a plan view. The vertical scanning circuit section 406 receives control pulses supplied from the control unit 403 and supplies control pulses to each pixel.

[0031] The vertical scanning circuit section 406 is composed of a shift register and an address decoder that connect multiple rows as a single unit, and achieves high-speed readout by reading multiple rows at once. In particular, in the case of an imaging device that digitally counts the number of photons arriving at the APD and outputs the count value as a digital signal converted by photoelectric conversion from the pixel, the operation of the counter circuit that digitally counts the number of photons takes time. Therefore, it is preferable to read multiple rows simultaneously for high-speed readout. That is, the vertical scanning circuit section 406, which functions as a readout circuit that reads pixel signals from pixels, simultaneously reads out the pixel signals from pixels included in the first row and the pixel signals from pixels included in the second row.

[0032] Furthermore, the control unit 403 sets threshold information, which serves as the judgment criterion, and exposure time information, which indicates the timing for performing the judgment, to the count judgment circuit described later. Note that the arrangement of the signal line 405, the readout circuit 402, and the output circuit 407 is not limited to Figure 4. For example, the signal line 405 may be arranged so as not to extend in the row direction, and the readout circuit 402 may be placed at the end of the signal line 405. Also, the function of the signal processing circuit 401 does not necessarily need to be provided in one for each photoelectric conversion unit; a single signal processing unit may be shared by multiple photoelectric conversion units, and sequential signal processing may be performed.

[0033] Figure 5 shows the equivalent circuits of pixel 301 and the corresponding signal processing circuit 401 in Figures 3 and 4. The APD501 generates charge pairs corresponding to incident light through photoelectric conversion. One of the two nodes of the APD501 is connected to a power line to which a drive voltage VL (first voltage) is supplied. The other of the two nodes of the APD501 is connected to a power line to which a drive voltage VH (second voltage), which is higher than voltage VL, is supplied. In Figure 5, one node of the APD501 is the anode and the other node is the cathode. The anode and cathode of the APD501 are supplied with a reverse bias voltage such that the APD501 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, and an avalanche current is generated. The APD501's operating mode is classified into two types depending on the value of the reverse bias voltage used to operate it. The two modes are Geiger mode, in which the anode and cathode voltage difference is greater than the breakdown voltage, and linear mode, in which the anode and cathode voltage difference is near or below the breakdown voltage. An APD operating in Geiger mode is called a SPAD (Single Photon Avalanche Diode). In the case of a SPAD, for example, the driving voltage VL (first voltage) is -30V and the driving voltage VH (second voltage) is 1V.

[0034] The quench element 502 is connected to a power line to which the drive voltage VH is supplied and to one of the nodes, either the anode or cathode, of the APD 501. The quench element 502 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to the APD 501 and thereby suppressing avalanche multiplication (quench operation). The quench element 502 also works to restore the voltage supplied to the APD 501 to the drive voltage VH by allowing current to flow to compensate for the voltage drop caused by the quench operation (recharge operation). In this invention, the quench element 502 is composed of a MOS transistor, and the on and off states of the quench element 502 are controlled by a control signal CLK connected to the gate of the quench element. The control signal CLK is controlled by the signal generation unit in the control unit 403.

[0035] The waveform shaping unit 510 shapes the voltage change at the cathode of the APD501 obtained when the APD501 detects a photon, and outputs a pulse signal. For example, an inverter circuit can be used as the waveform shaping unit 510. Figure 5 shows an example in which one inverter is used as the waveform shaping unit 510, but a circuit in which multiple inverters are connected in series may be used, or other circuits that have a waveform shaping effect may be used. Here, the structure for outputting a pulse in response to a photon incident on the APD501 as described above can be defined as the sensor unit.

[0036] The counter circuit 511 counts the pulse signal output from the waveform shaping unit 510 and holds the count value. When the control pulse RES is supplied via the drive line 514, the signal held by the counter circuit 511 is reset. Furthermore, when the control pulse STOP is supplied via the drive line 517, the counter circuit 511 continues to hold the count value until the control pulse RES is supplied.

[0037] The count determination circuit 512 receives the count value held by the counter circuit 511 via the drive line 516 and a pulse signal from the control unit 403 via the drive line 518. When the count determination circuit 512 receives the pulse signal, it compares the count value with a predetermined threshold, and if it determines that the count value exceeds the threshold, it supplies a control pulse STOP to the counter circuit 511 via the drive line 517. Furthermore, it outputs the determination result and the count value at the pulse timing to the selection circuit 513. The pulse signal timing chart will be described later with reference to Figure 7.

[0038] The selection circuit 513 receives a control pulse SEL from the vertical scanning circuit section 406 in Figure 4 via the drive line 515 (not shown in Figure 4) in Figure 5, which switches the electrical connection between the count determination circuit 512 and the signal line 405. The selection circuit 513 includes, for example, a buffer circuit for outputting a signal, and outputs the output signal from the pixel count determination circuit 512 to the signal line 405.

[0039] Furthermore, switches such as transistors may be placed between the quench element 502 and the APD 501, or between the photoelectric conversion unit 302 and the signal processing circuit 401, to switch the electrical connections. Similarly, the supply of the drive voltage VH or drive voltage VL to the photoelectric conversion unit 302 may be electrically switched using switches such as transistors.

[0040] Figure 6 schematically shows the relationship between the switch control signal CLK, the voltage at the input terminal node nodeA of the waveform shaping unit 510, the voltage at the output terminal node nodeB, and the count value of the counter circuit 511 in the photoelectric conversion element shown in Figure 5. In this invention, the switch refers to the MOS transistor that constitutes the quench element 502 in Figure 5. When the control signal CLK is high level, the drive voltage VH is less likely to be supplied to the APD501, and when the control signal CLK is low level, the drive voltage VH is supplied to the APD501. A high level control signal CLK is, for example, 1V, and a low level control signal CLK is, for example, 0V. When the control signal CLK is high level, the switch is off, and when the control signal CLK is low level, the switch is on. The resistance value of the switch when the control signal CLK is high level is higher than the resistance value of the switch when the control signal CLK is low level. When the control signal CLK is at a high level, even if avalanche multiplication occurs in the APD501, recharge operation is less likely to occur, and the voltage supplied to the APD501 becomes below the breakdown voltage of the APD501. Therefore, the avalanche multiplication operation in the APD501 stops.

[0041] At time t1, the control signal CLK changes from high to low, the switch turns on, and the APD501 recharge operation begins. This causes the cathode voltage of the APD501 to transition to a high level. Then, the APD501 enters a state where avalanche multiplication is possible due to the difference in voltages applied to the anode and cathode. The cathode voltage is the same as the voltage of node nodeA. Subsequently, when the cathode voltage transitions from low to high, at time t2, the voltage of node nodeA becomes greater than or equal to the judgment threshold. Here, the judgment threshold is a voltage value uniquely determined by the electrical characteristics of the waveform shaping unit 510. At this time, the pulse signal output from node nodeB inverts, from high to low. When recharging is complete, the APD501 is subjected to a voltage of (drive voltage VH - drive voltage VL). Subsequently, between time t2 and time t3, the control signal CLK becomes high, and the switch turns off.

[0042] Next, at time t3, when a photon is incident on APD501, avalanche multiplication occurs in APD501, an avalanche multiplication current flows through the quench element 502, and the cathode voltage drops. In other words, the voltage at node A drops. If the voltage at node A falls below the threshold during the voltage drop, the voltage at node B changes from a low level to a high level. That is, the portion of the output waveform at node A that exceeds the threshold is shaped by the waveform shaping unit 510 and output as a signal at node B. This is then counted by the counter circuit 511, and the count value of the counter signal output from the counter circuit 511 increases by 1 LSB.

[0043] Although photons are incident on APD501 between time t3 and time t4, the switch is off, and the voltage applied to APD501 is not a voltage difference that allows for avalanche multiplication. Therefore, the voltage level of node A does not exceed the threshold.

[0044] At time t4, the control signal CLK changes from high to low, and the switch turns on. Consequently, a current flows through node A to compensate for the voltage drop from the drive voltage VL, and the voltage at node A returns to its original voltage level. At this time, at time t5, the voltage at node A exceeds the threshold, so the pulse signal at node B inverts, changing from high to low.

[0045] At time t6, node A settles at its original voltage level, and the control signal CLK changes from a low level to a high level. In principle, the period during which the control signal CLK is low only needs to be longer than the period during which node A transitions from low to high level. In Figure 6, the period during which the control signal CLK is low is set to be the same as the period during which node A transitions from low to high level. This allows the frequency of the control signal CLK to be set higher, thereby reducing the effect of the "nonlinear relationship between the number of output signals and the number of input signals," which will be discussed later. Subsequently, as explained from time t1 to time t6, the voltages of each node and signal lines change in response to the control signal CLK and the incidence of photons. However, when the recharge frequency of the APD is controlled by the control signal CLK, the relationship between the number of output signals and the number of input signals is not linear. In this case, the number of input signals refers to the number of photons incident on the APD, and the number of output signals refers to the photon count value detected by the imaging device. In SPADs, when avalanche breakdown occurs, secondary photons are emitted, causing emission crosstalk with adjacent pixels. However, if the effects of emission crosstalk are ignored, the relationship between the number of output signals and the number of input signals can be theoretically derived. Specifically, when the number of input signals is Nph, the number of output signals is Nct, the frequency of the control signal CLK (number of CLKs per unit time) is f, and the exposure time is T, it is described by the following equation (1).

[0046]

number

[0047] Figure 7 is a timing chart illustrating the operation of the signal processing circuit 401. The exposure time for each pixel is set by the control unit 403 to a predetermined exposure time of T / (n to the power of m-1). T represents the maximum exposure time within one frame. m is any integer such that m≧1, but Figure 7 shows the timing chart when m is set to 1≦m≦4. A pulse that goes Hi only at the moment the exposure time determined by t=T / (n to the power of m-1) is reached is supplied to the drive line 518. At the moment of the four exposure times T / (n to the power of m-1) when m=1, 2, 3, and 4, the count value of the counter circuit 511 reaches a predetermined count threshold. At this time, the photoelectric conversion element 200 is switched from Geiger mode to linear mode and the APD 501 is put into sleep mode. Once the APD 501 is in sleep mode, the pulse signal from the waveform shaping unit 510 is not output, and the count of the counter circuit 511 is maintained as is. The count determination circuit 512 then outputs T / (n to the power of (m-1)), which represents the exposure time corresponding to the control pulse, and the count value to the selection circuit 513.

[0048] Figure 8 shows the relationship between the exposure time for each pixel 301 in the photoelectric conversion element 200 and the count value of the counter circuit 511. In Figure 8, as in Figure 7, the control unit 403 sets a predetermined exposure time T / (n to the power of (m-1)) for 1 ≤ m ≤ 4. When the upper limit of the count of the counter circuit 511 is Cmax, the count threshold is set to Cmax / n for reasons described later. In Figure 8, it is assumed that the count of a certain pixel increases in proportion to time. As described in Figure 7, the count determination circuit 512 determines whether the count value exceeds the count threshold at the moment of T / (n to the power of (m-1)) in order of increasing exposure time. As shown in Figure 8, when the count value increases, it is determined that the count threshold is not exceeded at the moments of T / n3 and T / n2, but is exceeded at the moment of T / n. At this time, the count is stopped at the moment of T / n, and the count value Cout at T / n and the exposure time T / n are output from the photoelectric conversion element 200 as the determination result of the determination circuit 512. While the exposure time T / n can be output directly as time information, as shown in Figure 9, the exposure time information Tcode to be output can be determined according to the exposure time when the count threshold is exceeded.

[0049] The exposure time information Tcode shown in Figure 9 is an example of the output data format. The count estimate value (Cest) shown in Figure 8 is calculated as Cest = Cout × n, assuming that the count increases at the same rate during the exposure time from 0 to T / n and the non-exposure time from T / n to T. In this count estimation method, as mentioned above, it is assumed that the count increases at the same rate during the exposure time from 0 to T / n and the non-exposure time from T / n to T. Therefore, if the upper count value is reached before the count threshold is reached, the rate of increase during the exposure time from 0 to T / n cannot be correctly estimated, and the accuracy of the count estimation decreases. Thus, in this count estimation method, in order to maintain the accuracy of the count estimation, a threshold determination is made before the upper count value (Cmax) is reached. As mentioned above, when the recharge frequency of the APD is controlled by the control signal CLK, the relationship between the number of output signals and the number of input signals is not linear, so linearity correction is necessary to further improve the accuracy of the count estimate value.

[0050] Figure 10 shows the relationship between the exposure time for each pixel 301 in the photoelectric conversion element 200 and the count value of the counter circuit 511. It shows how the count increases when the count value becomes equal to the count threshold at the moment of each exposure time for which threshold determination is performed. As mentioned above, in this count estimation method, in order to maintain the accuracy of the count estimation, threshold determination is performed before the upper limit of the count (Cmax) is reached. Therefore, if the timing of threshold determination when the count increases at a rate of increase that reaches Cmax at the moment of the maximum exposure time T is denoted as t1, then it is necessary to determine the timing of threshold determination t2 when the count increases at a rate of increase that reaches Cmax at the moment of t1. That is, when the count threshold is Cmax / n, the time until the count reaches Cmax / n when it increases at a rate of increase that reaches Cmax at the moment of the maximum exposure time T is calculated as T × (1 / n), so t1 = T / n. Similarly, t2 = t1 × (1 / n) = T / n2. In this way, each exposure time for which threshold determination is performed is calculated as T / (n to the power of (m-1)). As shown in Figure 5, when the APD recharge frequency is controlled by the control signal CLK, the relationship between the number of output signals and the number of input signals is not linear, as shown in equation (1). Therefore, linearity correction is performed based on the estimated count calculated by the signal processing circuit 401. Specifically, linearity correction refers to determining the number of input signals Nph from the number of output signals Nct per exposure time using the following equation (2), where f is the frequency of the control CLK (number of CLKs per unit time) and T is the length of the exposure time.

[0051]

number

[0052] The input signal number Nph derived by equation (2) in this invention is the number of photons per exposure time for threshold determination. Therefore, the number of photons in one frame exposure time is calculated as Nph × (n to the power of (m-1)).

[0053] Next, the configuration of the functions of the imaging device 100 in this embodiment will be described using Figure 11. In Figure 11, the optical lens 101, image sensor 102, and CPU 103 are connected to the internal bus 112. Furthermore, the internal functions of the CPU 103 include an optical lens control unit 1100, an image sensor control unit 1101, an exposure control unit 1102, a subject detection unit 1103, and an image processing unit 1104. That is, among the functions of each block in Figure 11, those that are implemented by software can be implemented by the CPU 103 executing a program stored in a non-volatile storage medium such as ROM 107. Each part connected to the internal bus 112 is configured to exchange data with each other via the internal bus 112. A description of the configuration similar to that in Figure 1 will be omitted.

[0054] The optical lens control unit 1100 transmits control signals to the optical lens 101 to adjust the aperture, focus, zoom, and other parameters.

[0055] The image sensor control unit 1101 transmits control signals to the image sensor 102 in order to set the exposure time of the image sensor, such as the shutter speed.

[0056] The exposure control unit 1102 transmits control signals related to exposure, such as aperture, shutter speed, and gain, to the optical lens control unit 1100, the image sensor control unit 1101, and the image processing unit 1104, in accordance with the shooting settings of the imaging device 100. In addition, when automatic exposure control is enabled, it detects the brightness of the image and automatically controls the aperture, shutter speed, gain, and other exposure-related settings to achieve the appropriate brightness.

[0057] The exposure control unit 1102 and the image sensor control unit 1101 receive information about pixels or regions identified by the identification means (image processing unit 1104, described later) and function as control means to change the imaging settings. Specifically, the control means dynamically changes the imaging settings to reduce the number of pixels or regions that have a difference between the first and second determination results obtained from multiple frames. This improves consistent image quality and the visibility of moving subjects.

[0058] The subject detection unit 1103 is a detection means that detects the presence or absence of a specific subject and the coordinate position of the subject within an image, based on the image captured by the image sensor 102 or the image processed by the image processing unit 1104. For example, it can detect people, cars, boats, etc., and may also detect multiple subjects.

[0059] The image processing unit 1104 performs various image processing on the image captured by the image sensor 102. The image processing unit 1104 functions as an acquisition means that obtains a first judgment result and a second judgment result from each of the multiple frames output from the image sensor 102. Specifically, the acquisition means temporarily stores the data output from the image sensor 102 in the frame memory 106 or RAM 108, and obtains the first judgment result and the second judgment result from the stored data. Each judgment result is information indicating whether the count value is above a predetermined threshold, based on the exposure time information for each frame.

[0060] The first and second determination results refer to the determination results by the count determination circuit 512 for frames acquired at different timings by the acquisition means. The first determination result is based on whether the count value in a frame at a certain time is above a predetermined threshold. On the other hand, the second determination result is based on whether the count value in a frame at a later time, different from the first determination result, is above a threshold. More specifically, the first and second determination results include the exposure time when the predetermined count threshold was reached for each pixel or region in the corresponding frame. As mentioned above, the count determination circuit 512 determines whether the count value has reached a predetermined count threshold at a plurality of pre-set exposure time timings (four exposure times in this embodiment). The count determination circuit 512 then outputs exposure time information Tcode and the count value when the predetermined count threshold is reached. Therefore, in each frame, the first and second determination results may include exposure time information Tcode when the count value reached the predetermined count threshold. The first and second determination results are output from the count determination circuit 512 for each pixel or region and can be acquired by the acquisition means. The exposure time can also be rephrased as the determination time. In other words, it can be interpreted as a determination time indicating the timing for determining whether the count value has reached a predetermined count threshold (in this embodiment, the count determination circuit 512 determines whether the count value has reached a predetermined count value at four determination times).

[0061] Furthermore, the image processing unit 1104 also functions as an identification means, comparing the first and second determination results acquired by the acquisition means to identify pixels or regions with differences. This algorithm treats the exposure time information of each frame as image data and calculates the difference. It identifies pixels or regions where the difference exceeds a predetermined threshold and changes the imaging settings based on that information. Here, a pixel or region with a difference between the first and second determination results is a pixel or region where the timing at which the count value reached a predetermined count threshold differs between the two frames. For example, consider a case where the exposure time information Tcode included in the first determination result for a certain pixel or region is 111, and the exposure time information Tcode included in the second determination result is 011. In this case, the first and second determination results for that pixel or region are different, which indicates that the exposure time information Tcode changed between the two frames. Therefore, it can be identified by the identification means as a pixel or region with a difference between the first and second determination results. However, the pixels or regions identified by the identification means are not limited to simply pixels or regions with different exposure time information Tcodes. For example, the exposure time information Tcode from the first determination result may be compared with the exposure time information Tcode from the second determination result, and if the difference between them is greater than or equal to a predetermined value, it may be identified as a pixel or region with a difference.

[0062] In addition to the functions described above, the image processing unit 1104 has functions equivalent to general image processing. For example, it can correct the amount of light in the peripheral areas of the image caused by the characteristics of the optical lens 101, correct sensitivity variations and defective pixels in the image sensor 102, and perform brightness-related corrections such as gain correction. It can also perform color-related corrections such as white balance correction and flicker correction.

[0063] Furthermore, it is possible to generate text strings and menus to inform the user of the settings status of the imaging device 100, and to overlay them onto images that have undergone various image processing. In addition to text information, it is also possible to overlay imaging assist displays such as histograms, vectorscopes, waveform monitors, zebra stripes, peaking, and false color.

[0064] Next, using Figures 12 to 15, we will explain the process of controlling the exposure time (threshold detection exposure time) for threshold determination of the image sensor 102 in response to a moving subject, which is performed by the imaging device 100. The flowchart in Figure 12 starts after the imaging device 100 is powered on and is executed repeatedly.

[0065] In the flowchart of Figure 12, first, in step S1201, the image processing unit 1104 acquires the first exposure time information (corresponding to the first determination result) output from the image sensor 102 and stores it in the frame memory 106 or RAM 108. In this embodiment, the exposure time information is output from the image sensor 102 for each pixel as exposure time information Tcode, as shown in Figure 9. Furthermore, because the data is output for each pixel, it can be treated as quadrant image data as shown in Figure 13.

[0066] Next, in step S1202, the image processing unit 1104 acquires second exposure time information output from the image sensor 102 and stores it in the frame memory 106 or RAM 108. The second exposure time information is obtained for frames at different times than the first exposure time information acquired in step S1201.

[0067] Next, in step S1203, the image processing unit 1104 refers to the first exposure time information obtained in step S1201 and the second exposure time information (corresponding to the second determination result) obtained in step S1202. It then determines whether there is a change in the exposure time information between frames at different times. In other words, the identification means (image processing unit 1104) identifies pixels or regions where the first determination result and the second determination result are different. If it is determined that there is a change, the process proceeds to step S1204; if it is determined that there is no change, the process proceeds to step S1206.

[0068] The method for determining changes in exposure time information involves treating the first and second exposure time information as image data as shown in Figure 13, calculating the number of pixels with a difference, and determining whether that number of pixels is greater than or equal to a predetermined number of pixels. In other words, a specific means is used to identify pixels or regions where the first determination result and the second determination result differ, and it is determined whether the identified number of pixels or regions is greater than or equal to a predetermined number of pixels or regions. For example, in the case of an image size of 1080 vertically x 1920 horizontally, the total number of pixels is 2,073,600, and it is determined whether there is a difference in 207,360 or more pixels, which is 10% of the total number of pixels. Alternatively, the first and second exposure time information may be updated at the same period, and changes may be determined using inter-frame differences. Alternatively, the first exposure time information may use exposure time information when there are no moving subjects, and the second exposure time information may be updated at a shorter period than the first exposure time information, and changes may be determined using background differences. Furthermore, the imaging settings may be changed based on the detection results from the subject detection unit 1103 and the pixels or regions identified by the identification means (image processing unit 1104). For example, the region in which differences are calculated may be limited to the region including the region in which the subject detection unit 1103 detected a subject, for the image processed by the image processing unit 1104. That is, pixels or regions with differences may be identified within the region including the region in which a subject was detected, and the imaging settings of the image sensor 102 may be changed to reduce the identified pixels or regions.

[0069] Next, in step S1204, the image processing unit 1104 calculates a threshold exposure time (determination time) using the first exposure time information acquired in step S1201 and the second exposure time information acquired in step S1202. Specifically, it determines a determination time such that no change occurs in the exposure time information Tcode between frames at different time points. In other words, it determines the determination time in such a way that it reduces the number of pixels or regions identified by the specific means.

[0070] Here, we will explain using the example where the first exposure time information is shown in Figure 14(a) and the second exposure time information is shown in Figure 14(b). First, we calculate the mode of the exposure time information for the pixels that have a difference between Figure 14(a) and Figure 14(b). In Figure 14(a), we extract the exposure time information difference pixel 1401 from the exposure time information 1400 and calculate the exposure time information of the mode of the exposure time information difference pixel 1401. Similarly in Figure 14(b), we extract the exposure time information difference pixel 1403 from the exposure time information 1402 and calculate the exposure time information of the mode. In the case of Figure 14(a), the mode of the first exposure time information difference pixel is 000, and in the case of Figure 14(b), the mode of the second exposure time information difference pixel is 011.

[0071] Next, a threshold exposure time is calculated from the mode of the difference pixels of the first exposure time information and the mode of the difference pixels of the second exposure time information such that no change occurs in the exposure time information. This is explained using Figure 15(a), which is the case when n=8 in Figure 9, as an example. In this case, the most frequent exposure time of the difference pixels of the first exposure time information is T, and the most frequent exposure time of the difference pixels of the second exposure time information is T / 64. Therefore, to make the difference pixels of the first exposure time information and the second exposure time information the same exposure time, n is changed to a value greater than 64 as shown in Figure 15(b), and the threshold exposure time corresponding to the exposure time information Tcode is calculated. Furthermore, the combination of threshold exposure times is not limited to the formula in Figure 9; any combination of threshold exposure times is acceptable as long as the difference pixels of the first exposure time information and the second exposure time information are the same, as shown in Figure 15(c).

[0072] Next, in step S1205, the image sensor control unit 1101 (control means) changes the threshold determination exposure time. Specifically, it sets the threshold determination exposure time calculated in step S1204 for the image sensor 102.

[0073] Next, in step S1206, the image processing unit 1104 determines whether to change the threshold exposure time. If a change is made, the process proceeds to step S1207; otherwise, the flowchart process ends. The method for determining whether to change the threshold exposure time may be, for example, whether a predetermined time or number of frames has elapsed since the threshold exposure time was changed in step S1205. For example, it may be determined that a predetermined time, such as 5 seconds, has elapsed, or that a predetermined number of frames, such as 150 frames, has elapsed. Alternatively, the number of pixels whose difference exceeds a predetermined threshold may be calculated using inter-frame difference or background difference on the captured image, and whether that number of pixels is below the predetermined threshold may be used as the determination condition. For example, in the case of an image size of 1080 vertically x 1920 horizontally, the total number of pixels is 2,073,600, and 207,360, which is 10% of the total number of pixels, is set as the threshold number of pixels, and it is determined whether the number of pixels determined to be differences is below the threshold.

[0074] Next, in step S1207, the image sensor control unit 1101 (control means) changes the threshold determination exposure time (i.e., changes the imaging setting of the image sensor 102). Specifically, it sets the threshold determination exposure time for the image sensor 102 that was set before the change in step S1205. The threshold determination exposure time that was set before the change in step S1205 may be stored in the RAM 108 and used, or it may be stored as an initial value in the ROM 107 and used.

[0075] In this embodiment, an example was described in which a first and second judgment result output from the count judgment circuit 512 are acquired in two frames output from the image sensor 102. On the other hand, one or more judgment results may be acquired in addition to the first and second judgment results. That is, pixels or regions in which the exposure time information Tcode differs among three or more judgment results may be identified by identification means.

[0076] As explained above, changes in exposure time information are determined from the exposure time information of frames at different times. Then, when a change in exposure time information occurs, a threshold exposure time (determination time) is set so that no further change in exposure time information occurs, thereby suppressing the decrease in visibility of moving subjects.

[0077] <Second Embodiment> The operation of the imaging device according to this embodiment will now be described. Since the configuration is the same as in the first embodiment, its description will be omitted. Figure 16 is a flowchart showing the operation of the imaging device according to this embodiment. In this flowchart, steps that perform the same processing as in the flowchart of Figure 12 are given the same step numbers as in Figure 12, and their explanations are omitted.

[0078] In the flowchart of Figure 16, steps S1201 to S1203 are the same as in Figure 12. If a change in exposure time information is detected in step S1203, the process proceeds to step S1601; otherwise, the process proceeds to step S1603.

[0079] Next, in step S1601, the image processing unit 1104 calculates an exposure compensation value using the first exposure time information acquired in step S1201 and the second exposure time information acquired in step S1202. Specifically, it calculates an exposure compensation value such that there is no change in the exposure time information between frames at different times.

[0080] Let's explain using the example where the first exposure time information is shown in Figure 14(a) and the second exposure time information is shown in Figure 14(b). Similar to step S1204, the mode of the difference pixels for the first exposure time information is 000, and the mode of the difference pixels for the second exposure time information is 011.

[0081] Next, an exposure compensation value is calculated from the mode of the difference pixels of the first exposure time information and the mode of the difference pixels of the second exposure time information so that no change occurs in the exposure time information. This will be explained using Figure 15(a), which is the case when n=8 in Figure 9, as an example. Similar to step S1204, the most frequent exposure time for the difference pixels of the first exposure time information is T, and the most frequent exposure time for the difference pixels of the second exposure time information is T / 64. Therefore, an exposure compensation value is calculated so that the amount of light incident on the image sensor 102 becomes 64 times or more, so that the difference pixels of the first exposure time information and the second exposure time information have the same exposure time. For example, if the shutter speed is set to 1 / 1024, the shutter speed setting should be longer than 1 / 16 in order to increase the amount of light incident by 64 times or more. Here, shutter speed has been given as an example, but exposure compensation values ​​may also be calculated using other shooting settings such as aperture.

[0082] Next, in step S1602, the exposure control unit 1102 changes the exposure settings. For example, if the shutter speed setting is to be changed, the image sensor control unit 1101 sets the shutter speed setting value calculated in step S1601 to the image sensor 102. Similarly, if the aperture setting is to be changed, the optical lens control unit 1100 sets the aperture setting value calculated in step S1601 to the optical lens 101.

[0083] Next, in step S1603, the image processing unit 1104 determines whether to change the exposure settings. If the exposure settings are to be changed, the process proceeds to step S1604; otherwise, the flowchart process ends. The method for determining whether to change the exposure settings is the same as in step S1206.

[0084] Next, in step S1604, the exposure control unit 1102 changes the exposure setting. Specifically, it reflects the exposure setting that was set before the change in step S1602. Also, if the automatic exposure control process is being executed in another process, the automatic exposure control process may be stopped in step S1602 and restarted in step S1604.

[0085] As explained above, by determining changes in exposure time information from exposure time information of frames at different times, and by changing the exposure settings so that no change in exposure time information occurs when a change in exposure time information is detected, it is possible to suppress the decrease in visibility of moving subjects.

[0086] <Third Embodiment> The operation of the imaging device according to this embodiment will now be described. Since the configuration is the same as in the first embodiment, its description will be omitted. Figure 17 is a flowchart showing the operation of the imaging device according to this embodiment. In this flowchart, steps that perform the same processing as in the flowchart of Figure 12 are given the same step numbers as in Figure 12, and their explanations are omitted.

[0087] In the flowchart of Figure 17, steps S1201 to S1203 are the same as in Figure 12. If a change in exposure time information is detected in step S1203, the process proceeds to step S1701; otherwise, the process proceeds to step S1702.

[0088] Next, in step S1701, the image sensor control unit 1101 changes the exposure settings. Specifically, the image sensor control unit 1101 changes the exposure setting for the image sensor 102 to one that does not include a threshold-determined exposure time. As a result, although it becomes impossible to secure a very wide dynamic range by using exposure with a threshold-determined exposure time, changes in exposure time will not occur in frames at different times.

[0089] Next, in step S1702, the image processing unit 1104 determines whether to change the exposure settings. If the exposure settings are to be changed, the process proceeds to step S1703; otherwise, the flowchart process ends. The method for determining whether to change the exposure settings is the same as in step S1206.

[0090] Next, in step S1703, the image sensor control unit 1101 changes the exposure setting. Specifically, the image sensor control unit 1101 changes the exposure setting for the image sensor 102 to include a threshold-determined exposure time. This makes it possible to capture images with a very wide dynamic range by using exposure with a threshold-determined exposure time.

[0091] As explained above, by determining changes in exposure time information from exposure time information of frames at different times, and by changing the exposure settings so that no change in exposure time occurs when a change in exposure time information is detected, it is possible to suppress the decrease in visibility of moving subjects.

[0092] <Other Embodiments> Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0093] Furthermore, the software program that realizes the functions of the above-described embodiment may be supplied directly from a computer-readable storage medium or to a system or device having a computer capable of executing the program using wired / wireless communication.

[0094] Therefore, in order to implement the functional processing of the present invention on a computer, the program code supplied to and installed on the computer itself also realizes the present invention. In other words, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

[0095] In that case, the form of the program is irrelevant, as long as it possesses the functionality of a program, including object code, programs executed by an interpreter, and script data supplied to the OS.

[0096] The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory.

[0097] Another possible method for supplying the program is to store the computer program forming the present invention on a server on a computer network, and then have connected client computers download and run the computer program.

[0098] The disclosures herein include the following determination devices, determination methods, programs, and storage media.

[0099] (Item 1) A sensor unit including an avalanche photodiode that emits pulses in response to photons incident on the avalanche photodiode, A counter circuit that counts the pulses emitted from the sensor unit, A determination circuit for determining whether the count value of the counter circuit is equal to or greater than a predetermined threshold, comprising a determination circuit for determining whether the count value is equal to or greater than the predetermined threshold in each of a set of determination times, An image sensor having an output circuit that outputs the count value and the determination result by the determination circuit, An acquisition means for acquiring the determination results of the determination circuit in two frames output from the image sensor, the first determination result and the second determination result corresponding to each of the two frames, The image sensor includes a means for identifying a pixel or region where there is a difference between the first determination result and the second determination result, An imaging apparatus characterized by having a control means for changing the imaging settings of the image sensor in order to reduce the pixels or regions identified by the identification means.

[0100] (Item 2) The determination circuit outputs the determination time in which the count value is determined to be equal to or greater than the predetermined threshold among the plurality of determination times. The first and second determination results include the determination time during which the count value output from the determination circuit is determined to be equal to or greater than a predetermined threshold. The imaging apparatus according to item 1, characterized in that the identifying means identifies pixels or regions where the determination time differs between the first determination result and the second determination result.

[0101] (Item 3) The imaging apparatus according to item 1, characterized in that the change in the imaging settings includes changing one or more of the multiple determination times.

[0102] (Item 4) The imaging device according to item 1, characterized in that the change in the imaging settings includes changing the shutter speed or aperture.

[0103] (Item 5) The imaging apparatus according to item 1, characterized in that the change in the imaging settings includes changing it so that the determination circuit does not perform determination.

[0104] (Item 6) The system further includes a detection means for detecting a subject from an image acquired from the sensor unit, The imaging apparatus according to item 1, characterized in that the imaging settings are changed based on the detection result by the detection means and the pixel or region identified by the identification means.

[0105] (Item 7) The imaging apparatus according to item 6, characterized in that the identifying means identifies pixels or regions in the image sensor from regions including regions detected by the detection means from each of the two frames, where the first determination result and the second determination result are different.

[0106] (Item 8) A sensor unit including an avalanche photodiode that emits pulses in response to photons incident on the avalanche photodiode, A counter circuit that counts the pulses emitted from the sensor unit, A determination circuit for determining whether the count value of the counter circuit is equal to or greater than a predetermined threshold, comprising a determination circuit for determining whether the count value is equal to or greater than the predetermined threshold in each of a set of determination times, A control method for an imaging device equipped with an image sensor having an output circuit that outputs the count value and the determination result by the determination circuit, An acquisition step of acquiring the determination results of the determination circuit for two frames output from the image sensor, wherein a first determination result and a second determination result corresponding to each of the two frames are acquired. The image sensor includes a process of identifying a pixel or region where there is a difference between the first determination result and the second determination result, A control method for an imaging device, comprising a control step of changing the imaging settings of the image sensor so as to reduce the pixels or regions identified by the specified step.

[0107] (Item 9) A program to cause a computer to execute the control method for the imaging device described in item 8.

[0108] (Item 10) A computer-readable storage medium on which the program described in item 9 is stored. [Explanation of symbols]

[0109] 100 Imaging device 101 Optical Lenses 102 Image sensor 103 CPU 104 Video output drive unit 105 Video terminal 106 frame memory 107 ROM 108 RAM 109 Operation section 110 Display drive unit 111 Display section 112 Internal bus 1100 Optical lens control unit 1101 Image sensor control unit 1102 Exposure Control Unit 1103 Subject detection unit 1104 Image Processing Unit

Claims

1. A sensor unit including an avalanche photodiode that emits pulses in response to photons incident on the avalanche photodiode, A counter circuit that counts the pulses emitted from the sensor unit, A determination circuit for determining whether the count value of the counter circuit is equal to or greater than a predetermined threshold, comprising a determination circuit for determining whether the count value is equal to or greater than the predetermined threshold in each of a set of determination times, An image sensor having an output circuit that outputs the count value and the determination result by the determination circuit, An acquisition means for acquiring the determination results of the determination circuit in two frames output from the image sensor, the first determination result and the second determination result corresponding to each of the two frames, The image sensor includes a means for identifying a pixel or region where there is a difference between the first determination result and the second determination result, An imaging apparatus characterized by having a control means for changing the imaging settings of the image sensor in order to reduce the pixels or regions identified by the identification means.

2. The determination circuit outputs the determination time in which the count value is determined to be equal to or greater than the predetermined threshold among the plurality of determination times. The first and second determination results include the determination time during which the count value output from the determination circuit is determined to be equal to or greater than a predetermined threshold. The imaging apparatus according to claim 1, characterized in that the identifying means identifies pixels or regions where the determination time differs between the first determination result and the second determination result.

3. The imaging apparatus according to claim 1, characterized in that the change in the imaging settings includes changing one or more of the multiple determination times.

4. The imaging apparatus according to claim 1, characterized in that the change in the imaging settings includes changing the shutter speed or aperture.

5. The imaging apparatus according to claim 1, characterized in that the change in the imaging settings includes changing the settings so that the determination circuit does not perform a determination.

6. The system further includes a detection means for detecting a subject from an image acquired from the sensor unit, The imaging apparatus according to claim 1, characterized in that the imaging settings are changed based on the detection result by the detection means and the pixel or region identified by the identification means.

7. The imaging apparatus according to claim 6, wherein the identifying means identifies pixels or regions in the image sensor that differ from the first determination result and the second determination result, from a region that includes a region detected by the detection means from each of the two frames.

8. A sensor unit including an avalanche photodiode that emits pulses in response to photons incident on the avalanche photodiode, A counter circuit that counts the pulses emitted from the sensor unit, A determination circuit for determining whether the count value of the counter circuit is equal to or greater than a predetermined threshold, comprising a determination circuit for determining whether the count value is equal to or greater than the predetermined threshold in each of a set of determination times, A control method for an imaging device equipped with an image sensor having an output circuit that outputs the count value and the determination result by the determination circuit, An acquisition step of acquiring the determination results of the determination circuit for two frames output from the image sensor, wherein a first determination result and a second determination result corresponding to each of the two frames are acquired. The image sensor includes a process of identifying a pixel or region where there is a difference between the first determination result and the second determination result, A control method for an imaging device, comprising a control step of changing the imaging settings of the image sensor so as to reduce the pixels or regions identified by the specified step.

9. A program for causing a computer to execute the control method of the imaging apparatus described in claim 8.

10. A computer-readable storage medium storing the program described in claim 9.

Images