Imaging device, control method thereof, program, and storage medium

Abstract

To simultaneously obtain a high frame rate moving image for recording or a high frame rate moving image for display while maintaining a continuous shooting speed when continuously shooting still images. [Solution] The device comprises a plurality of pixels, each pixel having a sensor means for outputting a pulse signal at a frequency corresponding to the frequency of receiving photons and a counting means for counting the number of said pulse signals; a control means for controlling the timing of resetting the count value counted by said counting means and the timing of reading out the count value; and a generation means for generating a first signal at a predetermined first frame rate based on the count values ​​read out at a plurality of different times and the timing at which the count value was read out, and for generating a second signal at a second frame rate corresponding to an exposure time longer than each frame period of the first frame rate.

Description

[Technical Field]

[0001] The present invention relates to an imaging device, a control method thereof, a program, and a storage medium, and more particularly to a technique for continuously capturing moving images and still images in parallel using an imaging device. [Background technology]

[0002] 2. Description of the Related Art Conventionally, imaging devices have been required to assist the photographer in framing by recording and displaying moving images at a frame rate higher than that of still images between continuous shooting of still images.

[0003] For example, Patent Document 1 discloses a technique for capturing still images during the interval between accumulation time and readout time while capturing moving images for live view (LV) display. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-131605 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology disclosed in Patent Document 1, continuous shooting of still images is performed between shooting of moving images for display, which places a limit on the storage time of still images. On the other hand, if the storage time of still images is secured beyond the limit, the frame rate of LV display will drop, and the frame rate of continuous shooting of still images will also drop.

[0006] The present invention has been made in consideration of the above problems, and aims to make it possible to simultaneously obtain high frame rate moving images for recording or for display while maintaining the continuous shooting speed when continuously shooting still images. [Means for solving the problem]

[0007] In order to achieve the above object, the imaging device of the present invention comprises a plurality of pixels, each pixel having a sensor means for outputting a pulse signal at a frequency corresponding to the frequency of receiving photons and a counting means for counting the number of the pulse signals; a control means for controlling the timing of resetting the count value counted by the counting means and the timing of reading out the count value; and a generation means for generating a first signal at a predetermined first frame rate based on the count values ​​read out at a plurality of different times and the timing at which the count values ​​were read out, and generating a second signal at a second frame rate corresponding to an exposure time longer than each frame period of the first frame rate. [Effects of the Invention]

[0008] According to the present invention, it is possible to simultaneously obtain high frame rate moving images for recording or for display while maintaining the continuous shooting speed when continuously shooting still images. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image sensor according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of the configuration of a sensor substrate according to an embodiment. [Figure 3] FIG. 2 is a diagram showing an example of the configuration of a circuit board according to an embodiment. [Figure 4] FIG. 2 is a block diagram showing a schematic configuration of a photoelectric conversion unit and a signal processing circuit according to the first embodiment. [Figure 5] FIG. 3 is a diagram schematically illustrating the relationship between the operation of a photoelectric conversion unit and an output signal. [Figure 6] FIG. 1 is a block diagram showing a schematic configuration of an imaging apparatus according to an embodiment. [Figure 7] FIG. 3 is a timing chart for explaining control of the image sensor according to the first embodiment. [Figure 8] 4 is a flowchart showing a method for driving an image sensor according to the first embodiment. [Figure 9]4 is a flowchart showing a method for driving an image sensor according to the first embodiment. [Figure 10] 4 is a flowchart of a moving image calculation process according to the first embodiment. [Figure 11] FIG. 10 is a timing chart for explaining a problem with exposure time in the modified example of the first embodiment. [Figure 12] FIG. 10 is a timing chart for explaining the timing of starting an exposure time and reading out a signal in a modified example of the first embodiment. [Figure 13] FIG. 10 is a block diagram showing a schematic configuration of a photoelectric conversion unit and a signal processing circuit according to a modification of the first embodiment. [Figure 14] FIG. 10 is a timing chart for explaining control of the image sensor according to the second embodiment. [Figure 15] FIG. 10 is a timing chart for explaining control of an image sensor according to a modified example of the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] First Embodiment 1 is a diagram showing an example of the configuration of an image sensor 100 according to an embodiment of the present invention. In this embodiment, the image sensor 100 has a so-called stacked structure in which two semiconductor substrates, a sensor substrate 11 and a circuit substrate 21, are stacked and electrically connected. However, the image sensor 100 may have a so-called non-stacked structure in which the components included in the sensor substrate and the components included in the circuit substrate are arranged on the same semiconductor layer. The sensor substrate 11 includes a photoelectric conversion region 12, and the circuit substrate 21 includes a signal processing region 22 that processes signals acquired by the photoelectric conversion region 12.

[0012] FIG. 2 is a diagram showing an example of pixel arrangement on the sensor substrate 11. The photoelectric conversion region 12 of the sensor substrate 11 includes a plurality of photoelectric conversion units 102 included in a plurality of pixels 101 arranged two-dimensionally in a plurality of rows and a plurality of columns. The photoelectric conversion units 102 are configured by avalanche photodiodes (APDs). The photoelectric conversion units 102 output a voltage that drops at a frequency corresponding to the frequency of receiving photons. Note that the number of pixels included in the photoelectric conversion region 12 and the number of rows and columns of the pixels are not particularly limited.

[0013] 3 is a diagram showing an example of the arrangement of each circuit on the circuit board 21. The circuit board 21 has a signal processing circuit 103 that processes the voltage output from each photoelectric conversion unit 102 shown in FIG. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114. Each pixel 101 includes a photoelectric conversion unit 102 and a signal processing circuit 103.

[0014] The vertical scanning circuit 110 receives control pulses supplied from a control pulse generating unit 115 and sequentially supplies the control pulses to a plurality of signal processing circuits 103 arranged in the row direction. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

[0015] In each pixel 101, the voltage output from the photoelectric conversion unit 102 is processed by the corresponding signal processing circuit 103. The signal processing circuit 103 includes a counter for counting the number of pulse signals generated in accordance with a drop in the voltage output from the photoelectric conversion unit 102, and a signal (count value) is output to a vertical signal line 113 from the signal processing circuit 103 of a row selected by the vertical scanning circuit 110. The readout circuit 112 incorporates a plurality of buffers connected to each vertical signal line 113, and holds the signals output to each vertical signal line 113. The horizontal scanning circuit 111 inputs control pulses that sequentially select each column to the readout circuit 112, and the signals held in the plurality of buffers of the horizontal scanning circuit 111 are output to the outside of the image sensor 100 via the output circuit 114.

[0016] By repeating the above-described row selection by the vertical scanning circuit 110 and column selection by the horizontal scanning circuit 111, signals for all pixels processed by the signal processing circuit 103 can be output.

[0017] 2 and 3, a plurality of signal processing circuits 103 are arranged in a region overlapping the photoelectric conversion region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the photoelectric conversion region 12 in a plan view. In other words, the sensor substrate 11 has a photoelectric conversion region 12 and a non-photoelectric conversion region arranged around the photoelectric conversion region 12. The vertical scanning circuit 110, the horizontal scanning circuit 111, the readout circuit 112, the output circuit 114, and the control pulse generating unit 115 are arranged in a region of the circuit substrate 21 overlapping the non-photoelectric conversion region in a plan view.

[0018] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 is not limited to the example shown in Fig. 3. For example, the vertical signal lines 113 may be arranged extending in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113. Furthermore, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit 102, and a configuration may be adopted in which one signal processing unit is shared by multiple photoelectric conversion units 102 and performs signal processing sequentially.

[0019] FIG. 4 is a block diagram showing the configuration of one photoelectric conversion unit 102 and one signal processing circuit 103 corresponding to the photoelectric conversion unit 102 in FIGS.

[0020] The APD 201 included in the photoelectric conversion unit 102 generates charges according to incident light through photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line that supplies a drive voltage VL. The other of the two nodes of the APD 201 is connected to a power supply line that supplies a drive voltage VH that is higher than the voltage VL.

[0021] In Figure 4, one node of the APD 201 is the anode, and the other node of the APD 201 is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. With this voltage supplied, the charge generated by incident light undergoes avalanche multiplication, generating an avalanche current.

[0022] When a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the voltage difference between the anode and cathode is close to or less than the breakdown voltage. APDs operating in Geiger mode are called SPADs (single photon avalanche diodes). For example, the drive voltage VL for a SPAD is -30V, and the drive voltage VH is 1V.

[0023] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, and a counter circuit 211. The quench element 202 is connected to a power supply line to which a drive voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0024] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to the APD 201 and suppressing avalanche multiplication (quench operation).The quench element 202 also functions to return the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0025] The waveform shaping unit 210 shapes the voltage change at the cathode of the APD 201 obtained when photons are detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. While Fig. 4 shows an example in which one inverter is used as the waveform shaping unit 210, a circuit in which multiple inverters are connected in series, or another circuit with a waveform shaping effect, may also be used.

[0026] The counter circuit 211 counts the number of pulse signals output from the waveform shaping unit 210 and holds the count value. Furthermore, when a control pulse φRES is supplied via a drive line 213, the signal held in the counter circuit 211 is reset. A control pulse φSEL is supplied via a drive line 214, switching between electrical connection / disconnection between the counter circuit 211 and the vertical signal line 113. When connected, the signal (count value) held in the counter circuit 211 of each pixel 101 is output to the vertical signal line 113.

[0027] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the drive voltage VH or the drive voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0028] 5 is a diagram schematically showing the relationship between the operation of the APD 201 and the output signal, with the input side of the waveform shaping unit 210 designated as node A and the output side as node B. Between time t0 and time t1, a potential difference of drive voltage VH-drive voltage VL occurs in the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at node A drops.

[0029] As the voltage drop increases further and the potential difference applied to APD 201 decreases, avalanche multiplication by APD 201 stops at time t2, and the voltage level at node A no longer drops below a certain value. Between time t2 and time t3, a current flows from voltage VL to node A to compensate for the voltage drop, and at time t3, node A stabilizes to its original potential level. At this time, the output waveform at node A is shaped by waveform shaping unit 210 into portions that exceed a predetermined threshold and portions that do not, and is output as a pulse signal to node B.

[0030] Next, the configuration of an imaging device 600 will be described as an example of a device using the above-described imaging element 100. FIG. 6 is a block diagram showing the configuration of the imaging device 600 according to the first embodiment. Note that the imaging device of the present invention includes any electronic device capable of incorporating an imaging function. Such electronic devices include video cameras, digital cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, drive recorders, wearable scopes, etc. These are merely examples, and the present invention can be applied to other electronic devices. The present invention can also be applied to a configuration in which the subject detection function and the imaging function are provided in separate devices capable of communicating with each other (for example, a main unit and a remote controller).

[0031] The imaging device 600 includes the imaging element 100 described with reference to Figures 1 to 5, an imaging optical system 601, an imaging signal processing circuit 602, an overall control and calculation unit 603, a read control circuit 604, a memory unit 605, a recording medium control interface (I / F) unit 606, a recording medium 607, a display unit 608, an external interface (I / F) unit 609, and an operation unit 610.

[0032] The imaging signal processing circuit 602 performs various image signal processing such as low-pass filtering to reduce noise, shading correction, and WB correction on the signal (count value) input from the imaging element 100. It also performs various corrections such as scratch correction, dark shading correction, and blackout correction, as well as compression, to generate image data.

[0033] The overall control and calculation unit 603 controls the entire image pickup device 600 and performs various calculations. The read control circuit 604 generates a drive pulse for driving the image pickup element 100 based on a control signal from the overall control and calculation unit 603.

[0034] The memory unit 605 temporarily stores image data generated by the imaging signal processing circuit 602. The recording medium control I / F unit 606 records and reads image data to a recording medium 607. The recording medium 607 is a removable storage medium such as a semiconductor memory, and records image data. The display unit 608 displays the image data. The external I / F unit 609 is an interface for communicating with an external computer or the like.

[0035] The operation unit 610 is used to input various user instructions to the imaging device 600 and is composed of various operation members such as buttons, dials, switches, a touch panel, a gaze detection device, and a sound detection device. Information about the operating conditions of the imaging device 600 set by the user through operation of the operation unit 610 is sent to the overall control calculation unit 603, which controls the entire imaging device 600 based on this information. The operation unit 610 includes a power button, a still image recording button, and a video recording button (not shown), and the imaging device 600 can be turned on and off by turning the power button on and off. Furthermore, when the imaging device 600 is on, turning on the still image recording button can instruct the start of still image recording, and turning on the video recording button can instruct the start of video recording. Note that a common member can be used as the still image recording button and the video recording button.

[0036] FIG. 7 is a timing diagram illustrating a control method for the image sensor 100 controlled by the overall control calculation unit 603 according to this embodiment. Here, as an example, control will be described for a case where the operation unit 610 instructs continuous shooting of still images at 30 fps while moving images for live view (LV) display (hereinafter referred to as "LV images") are being captured at 120 fps. In this case, the exposure time (count period) of the still images is set to an integer multiple of one frame period of the LV images, and the timing at which the counter circuit 211 starts counting and the timing at which the count value is read are controlled to match the frame rate of the LV images. In this embodiment, one still image is captured using an exposure time (25 ms) equivalent to three frame periods of the LV images. That is, as shown in FIG. 7, one frame period of the still images (T0 to T4) is evenly divided to form one frame period of the LV images. In this case, one frame period of the LV images is 8.33 ms (≈33.3 ms / 4).

[0037] In addition, in FIG. 7, the frame period of each LV image is represented as LV frame i_j, where i (≧0) represents the number of the still image, and j (=1 to 4) represents the number of the LV image in one frame period of the still image.

[0038] The LV image and the still image do not necessarily need to have the same number of pixels, and for example, the LV image may be thinned out and read out at a predetermined rate (for example, every third pixel) in both the horizontal and vertical directions. In this case, the signal read out in LV frame i_4 serves the roles of both the display moving image and the still image, but they may be read out according to their intended use, for example, by first reading out signals every third pixel for the LV image, and then reading out signals for all pixels for the still image after that reading is complete.

[0039] Furthermore, the read control circuit 604 may have a power saving function that turns off the power supply to the read circuit during periods when readout is not being performed. This corresponds to the readout circuit power saving period shown in FIG.

[0040] 7, the exposure time indicates the exposure time corresponding to LV frames 1_1 to 1_4 in the frame period of still image 1. LV frame 1_1 has an exposure time from frame start time T0 of still image 1 to end time T1 of LV frame 1_1. LV frame 1_2 has an exposure time from time T1 to time T2, LV frame 1_3 has an exposure time from time T1 to time T3, and LV frame 1_4 has an exposure time from time T1 to time T4. That is, the counter circuit 211 is reset at times T0 and T1, and count values ​​C1_1, C1_2, C1_3, and C1_4 are read out from the counter circuit 211 at times T1 to T4, respectively.

[0041] The signals counted during the period of LV frame 1_1 are read out between times T1 and T2 and are promptly processed by the imaging signal processing circuit 602. Similarly, the signals counted during the periods of LV frame 1_2, LV frame 1_3, and LV frame 1_4 are sequentially read out between times T2 and T3, between times T3 and T4, and between times T4 and T5, respectively, and are promptly processed by the imaging signal processing circuit 602.

[0042] The overall control calculation unit 603 calculates differences between signals where the exposure times of LV images overlap in order to keep the exposure times of LV images constant, and controls the display unit 608 to display the resulting signals. For example, count value C1_3 corresponds to an exposure time for two frame periods, LV frame 1_2 and LV frame 1_3, and by calculating the difference between count value C1_2 corresponding to LV frame 1_2, the count value of the exposure time corresponding to LV frame 1_3 can be obtained. Similarly, count value C1_4 corresponds to an exposure time for three frame periods, LV frame 1_2, LV frame 1_3, and LV frame 1_4, and by calculating the difference between count value C1_3 and count value C1_4, the count value of the exposure time corresponding to LV frame 1_4 can be obtained. At this time, the overall control calculation unit 603 performs image processing so that the luminance between the frames of each LV image is constant.

[0043] 8, 9, and 10 are flowcharts showing details of a method for driving the image sensor 100 in this embodiment. Note that the operations of the steps shown in the flowcharts of FIGS. 8, 9, and 10 are performed by a CPU or the like serving as a computer in the overall control and calculation unit 603 executing a computer program stored in memory.

[0044] First, in S101 of Fig. 8, it is determined whether or not to start LV display on the display unit 608. The determination of S101 is repeated until LV display is started (No in S101). When LV display is started (Yes in S101), the process proceeds to S102, where the count value of the counter circuit 211 is reset by the control pulse φRES. This operation corresponds to the reset operation of the counter circuit 211 at time T0 or time T1 in Fig. 7. As a result, counting by the counter circuit 211 is started in S103.

[0045] Next, in S104, it is determined whether the current counting operation, i.e., the exposure time of the image sensor 100, serves as both the exposure time for the still image and the exposure time for the LV image. If it is determined that it serves as both (Yes in S104), the process proceeds to the processing of the flowchart in FIG. 9. If it is determined that it does not serve as both the exposure time for the still image and the exposure time for the LV image (No in S104), it is determined in the following S105 whether a predetermined time (one frame period of the LV image) has elapsed since the counting by the counter circuit 211 was started in S103. If one frame period of the LV image has not elapsed (No in S105), the process returns to S105. If it is determined that one frame period of the LV image has elapsed (Yes in S105), the counting by the counter circuit 211 is ended in the following S106, the count value is read out in S107, and an LV image based on the count value is output to the display unit 608 in S108.

[0046] In S109, it is determined whether or not to end the LV display. If it is determined that the LV display should be continued (No in S109), the process returns to S102, and if it is determined that the LV display should be ended (Yes in S109), this process ends.

[0047] Next, with reference to FIG. 9, an operation when it is determined in S104 that the current counting operation serves as both the exposure time for the still image and the exposure time for the LV image will be described.

[0048] First, in S201, the variable i is set to 0. In the following S202, it is determined whether one frame period of the LV image has elapsed since the counter circuit 211 started counting in S103. If one frame period of the LV image has not elapsed (No in S202), the process returns to S202. If it is determined that one frame period of the LV image has elapsed (Yes in S202), it is determined in the following S203 whether the exposure time of the still image has elapsed.

[0049] If it is determined that the exposure time for the still image has not elapsed, the process proceeds to S208, and the count value Count i is read out and output from the output circuit 114 to the imaging signal processing circuit 602. Thereafter, in S209, the variable i is incremented by 1, and the process proceeds to S210. In S210, the read count value Count i The process of displaying the LV image is performed using the above, and the process performed here will be described later with reference to Fig. 10. Note that the order of the process of S209 and the process of S210 may be reversed, or they may be performed in parallel.

[0050] After S210, the process returns to S202. At this time, in S202, it is determined whether one frame period of the LV image has elapsed since it was determined in the previous S202 that one frame period of the LV image has elapsed, rather than since the counting by the counter circuit 211 was started in S103. Thereafter, the above-described process is repeated.

[0051] On the other hand, if it is determined in S203 that the exposure time for the still image has elapsed (Yes in S203), the counting by the counter circuit 211 is terminated in S204, and the count value Count i The count value Count is read out and output from the output circuit 114 to the image pickup signal processing circuit 602.i In the next step S206, this signal is used in the process of displaying an LV image, and in step S207, it is processed by the imaging signal processing circuit 602 and stored as an image signal of a still image in the memory unit 605. The process performed in step S206 will be described with reference to FIG.

[0052] After S207, in S211, it is determined whether still images should be continuously captured (continue continuous shooting). Here, for example, if continuous shooting mode is set by the operation unit 610 and the still image recording button is pressed, it is determined that still images should be continuously captured. If still images should be continuously captured, the process returns to S102 in Fig. 8, and if not, the process proceeds to S109 in Fig. 8.

[0053] Next, the LV image display process performed in S207 or S210 in Fig. 9 will be described with reference to Fig. 10. This process is performed when acquiring a signal for an LV image during the exposure time of a still image. As explained in Fig. 7, for readouts in which the exposure times of the LV image and the still image overlap, it is necessary to calculate the difference between the read count values.

[0054] In S301, it is determined whether the variable i is 0. If the variable i is 0 (Yes in S301), the count value Count i If the variable i is not 0 (No in S301), the count value Count is output to the display unit 608 in S303. i and the count value of the previous LV frame, Count iー1 The difference between this count value and the count value read out one frame period before is output to the display unit 608. This removes the count values ​​of overlapping exposure times, and makes it possible to display an image based on the count values ​​corresponding to each frame period of the LV image. When the process of S302 or S303 is completed, the process returns to S207 or S210 in FIG.

[0055] As described above, according to the first embodiment, by reading out an image for LV display during exposure of a still image, it is possible to generate an LV image even during exposure of a still image, and it is also possible to maintain constant the frame rate of LV display and the frame rate for continuous shooting of still images.

[0056] <Modification> In the first embodiment described above, counting is started simultaneously for all pixels, and count values ​​are read out row by row. Fig. 11 is a diagram showing the exposure time for each row of frames 1_2 and 1_3 in Fig. 7. As shown in Fig. 11, exposure of all pixels is started by resetting the count value at the start of LV frame 1_2 (time T1), but reading of LV frame 1_2 is performed row by row from time T2. This results in a difference in exposure time between the first row read out (the first row) and the last row read out (the Nth row).

[0057] To solve the above-mentioned problem, the timing at which counting starts may be shifted for each row depending on the readout. Fig. 12 is a diagram showing the exposure time for each row of LV frame 1_2 and LV frame 1_3. In this case, by shifting the timing at which the count value is reset to match the timing at which the count value is read out for each row, the exposure time for the first readout row (row 1) and the last readout row (row N) can be made the same.

[0058] Furthermore, the image sensor 100 shown in the first embodiment may be configured such that each signal processing circuit 103 has a memory. Fig. 13 is a block diagram showing the configuration of one photoelectric conversion unit 102 and one signal processing circuit 103 corresponding to the photoelectric conversion unit 102 when a memory circuit 212 is added to the signal processing circuit 103. Note that the same reference numerals are used to designate the same components as those shown in Fig. 4.

[0059] 13, a memory circuit 212 is configured subsequent to a counter circuit 211. A control pulse φRES is supplied to the counter circuit 211 via a drive line 213, and a control pulse φSEL is supplied to the memory circuit 212 via a drive line 214 and a control pulse φMEM is supplied to the memory circuit 212 via a drive line 215. The counter circuit 211 resets its count value when the control pulse φRES is supplied and starts counting from that point. When the control pulse φMEM is supplied, the memory circuit 212 acquires and holds the count value at that time from the counter circuit 211. Thereafter, when the control pulse φSEL is supplied, the memory circuit 212 outputs the held count value to the vertical signal line 113. This makes it possible to synchronize the exposure timing of the row to be read out first (the first row) and the row to be read out last (the Nth row).

[0060] <Second embodiment> A second embodiment of the present invention will be described below. Note that the image sensor and image sensor device in the second embodiment can use the image sensor 100 and image sensor device 600 described in the first embodiment or its modified example, and therefore a description thereof will be omitted here. In the second embodiment, the driving of the image sensor 100 is controlled by a method different from that in the first embodiment, and therefore a control method for the image sensor 100 in the second embodiment will be described below.

[0061] 14 is a timing chart for explaining a control method of the image sensor 100 controlled by the overall control calculation unit 603 according to the second embodiment. In the second embodiment, an operation in which the counter circuit 211 is reset at the readout timing of an LV frame is shown.

[0062] In this embodiment, as in the first embodiment, control will be described when continuous shooting of still images at 30 fps is instructed by the operation unit 610 while LV images are being shot at 120 fps. Also in the second embodiment, the exposure time (count period) of the still images is set to be an integer multiple of one frame period of the LV images, and the count start timing of the counter circuit 211 and the read timing of the count value are controlled to match the frame rate of the LV images. Furthermore, in this embodiment, one still image is shot with the exposure time (25 ms) of the still image being three frame periods of the LV images.

[0063] 14, in the second embodiment, the counter circuit 211 is reset at the read timing (times T0, T1, T2, T3, ...) for each LV frame. Then, at times T1 to T4, count values ​​C1_1, C1_2, C1_3, and C1_4 for LV frame 1_1, LV frame 1_2, LV frame 1_3, and LV frame 1_4, respectively, are read out from the counter circuit 211. Then, a still image is generated by adding up the count values ​​C1_2, C1_3, and C1_4 corresponding to the exposure times (frames 1_2 to 1_4).

[0064] As described above, according to the second embodiment, by adding a signal for an LV image to realize the exposure time for a still image, it is possible to capture a still image that can be accumulated in excess of the counter number of the counter circuit 211 without reducing the frame rate of the LV image.

[0065] In the first and second embodiments described above, the frame period of the LV image is ¼ of one frame period of a still image, and four readouts are performed, but the present invention is not limited to this. For example, the frame rate of the LV image may be changed when an instruction to capture a still image is received from the operation unit 610. Furthermore, the number of frames may be set according to the recognition accuracy in the imaging signal processing circuit 602, and one frame period of the LV image may be set to, for example, ⅕ or ⅓ of one frame period of a still image. Alternatively, the exposure time of the LV image may be changed according to the brightness of the subject.

[0066] Furthermore, display control may be performed to increase the display frame rate of the display unit 608 in accordance with the changed frame rate of the LV image, or multiple consecutive LV images may be added together to leave the display frame rate of the display unit 608 unchanged. Increasing the frame rate of the LV image allows for more precise control of the exposure time of the still image.

[0067] Furthermore, even if the exposure time of the LV image is set to 1 / 4 of one frame period of a still image, the exposure times of LV frames 1_1, 1_2, 1_3, and 1_4 may be set shorter depending on the brightness of the subject, image recognition accuracy, etc. In this case, for frames such as LV frame 1_1 that do not share the exposure time with a still image, the effective exposure time can be made shorter than one frame period of the LV image by resetting the counter circuit at the timing of the start of accumulation. Also, for frames such as LV frames 1_2, 1_3, and 1_4 that share the exposure time with a still image, the effective exposure time can be made shorter than one frame period of the LV image by performing a signal readout process at the timing of the start of exposure and subtracting the readout signal.

[0068] In the above-described first and second embodiments, still images are continuously shot between the shooting of LV images, but the present invention can also be applied to the case where still images are continuously shot between the shooting of moving images for recording.

[0069] <Modification> In the second embodiment, the exposure time of a still image is set to an integral multiple of one frame of a displayed moving image. However, this is not limited to this, depending on the brightness of the subject, the image recognition accuracy, etc. For example, if a signal is read out midway through an LV frame, it is possible to obtain a still image with the desired exposure time from the difference with that image. The operation at this time is shown in FIG. 15.

[0070] 15 shows control when the exposure time of a still image is set from the middle of LV frame 1_2 to the middle of LV frame 1_4. Here, the count value C1_2' at the timing when you want to start storing the still image is read out, and the count value C1_4' at the timing when you want to end the storage is read out. In this case, the still image can be obtained by adding the difference between C1_2 and C1_2', C1_3, and C1_4'.

[0071] Note that by introducing a similar concept into the first embodiment and subtracting the signal read out during the frame period of the LV image, the exposure time of the still image can be freely set even in the control of the first embodiment, without being limited to an integer multiple of the frame period of the LV image.

[0072] <Other embodiments> The present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device.

[0073] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0074] <Summary> The disclosure of this embodiment includes the following configuration.

[0075] (Item 1) a plurality of pixels, each pixel including a sensor means for outputting a pulse signal at a frequency corresponding to a frequency of receiving a photon, and a counting means for counting the number of the pulse signals; a control means for controlling the timing of resetting the count value counted by the count means and the timing of reading out the count value; a generating means for generating a first signal at a predetermined first frame rate based on count values ​​read out at different times and the times at which the count values ​​are read out, and generating a second signal at a second frame rate corresponding to an exposure time longer than each frame period of the first frame rate; An imaging device comprising: (Item 2) 2. The imaging device according to item 1, wherein the control means, when resetting the count value, performs control such that the count value is read out and then reset. (Item 3) 3. The imaging device according to item 1 or 2, wherein the control means controls so that the count value is reset at the start of each frame period when the exposure time does not occupy the frame period. (Item 4) the exposure time is an integer multiple of each frame period, The control means controls so as to reset the count value at the start of the exposure time. 4. The imaging device according to item 2 or 3. (Item 5) 5. The imaging device according to item 4, wherein the control means controls so as to read out the count value at the end of the exposure time. (Item 6) the control means controls the count means to read out a count value at the start and end of the exposure time, The generating means generates the second signal by subtracting the count value read at the start of the exposure time from the count value read at the end of the exposure time. 4. The imaging device according to item 3, (Item 7) the control means controls to read out the count value at the end of each frame period; When the exposure time is in each frame period, the generating means generates the first signal by subtracting the count value read out one frame period before from the count value read out at the end of each frame period. 7. The imaging device according to any one of items 3 to 6, (Item 8) The control means controls the counting means so as to reset the counting means at the start of each frame period. 3. The imaging device according to item 1 or 2, characterized in that: (Item 9) the exposure time is an integer multiple of each frame period, the control means controls to read out the count value at the end of each frame period; The generating means generates the second signal by adding up the count values ​​read out at the end of each frame period corresponding to the exposure time. 9. The imaging device according to item 8, (Item 10) the control means controls the count means to read out a count value at the start and end of the exposure time, After the exposure time is started, the generating means generates the second signal by subtracting the count value read at the start of the exposure time from a value obtained by adding the count value read at the end of each frame period from the start of the exposure time to the end of the exposure time and the count value read at the end of the exposure time. 9. The imaging device according to item 8, (Item 11) further comprising a setting means for setting an exposure time of the first signal; when the exposure time of the first signal is shorter than each of the frame periods, the control means controls so as to read out the count values ​​at the start and end of the exposure time of the first signal in each of the frame periods; 11. The imaging device according to any one of items 1 to 10, wherein the generating means generates the first signal by subtracting a count value read at the start of an exposure time of the first signal from a count value read at the end of an exposure time of the first signal. (Item 12) Item 12. The imaging device according to item 11, wherein the start of the exposure time of the first signal is the start of each frame period, or the end of the exposure time of the first signal is the end of each frame period. (Item 13) 13. The imaging device according to any one of items 1 to 12, further comprising an image processing unit that performs image processing so that the luminance between frames of the first signal generated by the generating unit is constant. (Item 14) 8. The imaging device according to item 5 or 7, characterized in that, when reading out the count value except at the end of the exposure time, pixels are thinned out from the plurality of pixels at a predetermined rate and the count value is read out. (Item 15) 15. The imaging device according to any one of items 1 to 14, further comprising a display control means for controlling the first signal to be displayed on a display means at the first frame rate. (Item 16) the plurality of pixels are two-dimensionally arranged in a plurality of rows and a plurality of columns, The imaging device described in any one of items 1 to 15, characterized in that the control means shifts the timing of the reset and the timing of the readout for each row so that the time from resetting the counting means to reading out the count value is constant. (Item 17) Each of the pixels further includes a memory for storing a count value counted by the counting means, 17. The imaging device according to any one of items 1 to 16, wherein the control means resets the plurality of pixels at the same timing and stores the count values ​​in the memory at the same timing for the plurality of pixels. (Item 18) further comprising a read means for reading the count value, 18. The imaging device according to any one of items 1 to 17, wherein the operation of the readout means is stopped while the count value is not being readout. (Item 19) an operation means for setting the exposure time and the second frame rate; a setting means for setting the first frame rate to be an integer multiple of the second frame rate; 19. The imaging device according to any one of items 1 to 18, further comprising: (Item 20) 20. The imaging device according to any one of items 1 to 19, wherein the first signal is a signal of a moving image for display or a moving image for recording, and the second signal is a signal of a still image. (Item 21) A control method for an imaging device including an imaging element having a plurality of pixels, each pixel including a sensor means for outputting a pulse signal at a frequency corresponding to a frequency of receiving photons, and a counting means for counting the number of the pulse signals, a control step in which a control means controls the timing of resetting the count value counted by the counting means and the timing of reading out the count value; a generating step in which a generating means generates a first signal at a predetermined first frame rate based on count values ​​read out at different times and the times at which the count values ​​are read out, and generates a second signal at a second frame rate corresponding to an exposure time longer than each frame period of the first frame rate; 10. A method for controlling an imaging device, comprising: (Item 22) 22. A program for causing a computer to execute each step of the method for controlling an imaging device according to Item 21. (Item 23) A program for causing a computer to function as a control means and a generation means of the imaging device according to any one of items 1 to 20. (Item 24) 24. A computer-readable storage medium storing the program according to item 22 or 23.

[0076] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0077] 11: sensor substrate, 12: photoelectric conversion region, 21: circuit board, 22: signal processing region, 100: imaging element, 101: pixel, 102: photoelectric conversion section, 103: signal processing circuit, 110: vertical scanning circuit, 111: horizontal scanning circuit, 112: readout circuit, 113: vertical signal line, 114: output circuit, 115: control pulse generation section, 201: avalanche photodiode, 202: quenching element, 210: waveform shaping section, 211: counter circuit, 212: memory circuit, 600: imaging device, 601: imaging optical system, 602: imaging signal processing circuit, 603: overall control and calculation section, 604: readout control circuit, 605: memory section, 606: storage medium control I / F section, 607: recording medium, 608: display section, 609: external I / F section, 610: operation section

Claims

1. a plurality of pixels, each pixel including a sensor means for outputting a pulse signal at a frequency corresponding to a frequency of receiving a photon, and a counting means for counting the number of the pulse signals; a control means for controlling the timing of resetting the count value counted by the count means and the timing of reading out the count value; a generating means for generating a first signal at a predetermined first frame rate based on count values ​​read out at different times and the times at which the count values ​​are read out, and for generating a second signal at a second frame rate corresponding to an exposure time longer than each frame period of the first frame rate; An imaging device comprising:

2. 2. The imaging device according to claim 1, wherein the control means, when resetting the count value, performs control so that the count value is read out and then reset.

3. 2. The imaging device according to claim 1, wherein the control means performs control so that the count value is reset at the start of each frame period when the exposure time does not occupy the frame period.

4. the exposure time is an integer multiple of each frame period, The control means controls so as to reset the count value at the start of the exposure time.

3. The imaging device according to claim 2.

5. 5. The imaging apparatus according to claim 4, wherein the control means controls the count value to be read out when the exposure time ends.

6. the control means controls the count means to read out a count value at the start and end of the exposure time, The generating means generates the second signal by subtracting the count value read at the start of the exposure time from the count value read at the end of the exposure time.

4. The imaging device according to claim 3.

7. the control means controls to read out the count value at the end of each frame period; When the exposure time is long during each frame period, the generating means generates the first signal by subtracting the count value read out one frame period before from the count value read out at the end of each frame period.

4. The imaging device according to claim 3.

8. The control means controls the counting means so as to reset the counting means at the start of each frame period.

2. The imaging device according to claim 1.

9. the exposure time is an integer multiple of each frame period, the control means controls to read out the count value at the end of each frame period; The generating means generates the second signal by adding up the count values ​​read out at the end of each frame period corresponding to the exposure time.

9. The imaging device according to claim 8.

10. the control means controls the count means to read out a count value at the start and end of the exposure time, After the exposure time is started, the generating means generates the second signal by subtracting the count value read at the start of the exposure time from a value obtained by adding the count value read at the end of each frame period from the start of the exposure time to the end of the exposure time and the count value read at the end of the exposure time.

9. The imaging device according to claim 8.

11. further comprising a setting means for setting an exposure time of the first signal; when the exposure time of the first signal is shorter than each of the frame periods, the control means controls so as to read out the count values ​​at the start and end of the exposure time of the first signal in each of the frame periods; 2. The imaging device according to claim 1, wherein the generating means generates the first signal by subtracting a count value read at the start of an exposure time of the first signal from a count value read at the end of an exposure time of the first signal.

12. 12. The imaging device according to claim 11, wherein the start of an exposure time of the first signal is the start of each frame period, or the end of an exposure time of the first signal is the end of each frame period.

13. 2. The imaging device according to claim 1, further comprising image processing means for performing image processing so that the luminance between frames of the first signal generated by the generating means is constant.

14. 8. The imaging device according to claim 5, wherein when reading out the count value except at the end of the exposure time, pixels are thinned out from among the plurality of pixels at a predetermined rate and the count value is read out.

15. 2. The imaging apparatus according to claim 1, further comprising display control means for controlling the first signal to be displayed on a display means at the first frame rate.

16. the plurality of pixels are two-dimensionally arranged in a plurality of rows and a plurality of columns, 2. The imaging device according to claim 1, wherein the control means shifts the timing of resetting the counting means and the timing of reading out the count value for each row so that the time from resetting the counting means to reading out the count value is constant.

17. Each of the pixels further includes a memory for storing a count value counted by the counting means, 2. The imaging device according to claim 1, wherein the control means resets the plurality of pixels at the same timing and causes the count values ​​of the plurality of pixels to be stored in the memory at the same timing.

18. further comprising a read means for reading the count value, 2. The imaging device according to claim 1, wherein the operation of said readout means is stopped while the count value is not being read out.

19. an operation means for setting the exposure time and the second frame rate; a setting means for setting the first frame rate to be an integer multiple of the second frame rate; 2. The imaging device according to claim 1, further comprising:

20. 2. The imaging device according to claim 1, wherein the first signal is a signal of a moving image to be displayed or a moving image to be recorded, and the second signal is a signal of a still image.

21. A control method for an imaging device including an imaging element having a plurality of pixels, each pixel including a sensor means for outputting a pulse signal at a frequency corresponding to a frequency of receiving photons, and a counting means for counting the number of the pulse signals, a control step in which a control means controls the timing of resetting the count value counted by the counting means and the timing of reading out the count value; a generating step in which a generating means generates a first signal at a predetermined first frame rate based on count values ​​read out at different times and the times at which the count values ​​are read out, and generates a second signal at a second frame rate corresponding to an exposure time longer than each frame period of the first frame rate; 10. A method for controlling an imaging device, comprising:

22. A program for causing a computer to execute each step of the method for controlling an imaging apparatus according to claim 21.

23. A program for causing a computer to function as the control means and the generation means of the imaging device according to claim 1.

24. A computer-readable storage medium storing the program according to claim 22 or 23.

Images