Imaging device, control method thereof, program, and storage medium

Abstract

An imaging device capable of acquiring subject luminance information during exposure while suppressing deterioration in image quality of a recorded image. [Solution] The image sensor includes an image sensor having a plurality of pixels arranged two-dimensionally, each pixel having a photoelectric conversion unit that emits pulses at a frequency corresponding to the frequency of photon reception, a counter that counts the number of pulses, and a memory that stores the counter's count value; an image processing unit that generates an image signal based on the counter's count value at the end of an exposure period; a display control unit that displays information using the image signal on a display unit; and a control unit that causes the image sensor to perform exposure for a first exposure period and an exposure for a second exposure period that includes the first exposure period but is longer than the first exposure period in a single shooting, and controls the display control unit to display information using the first image signal generated during the first exposure period on the display unit until the end of the second exposure period.

Description

[Technical Field]

[0001] The present invention relates to an imaging apparatus and a control method thereof. [Background technology]

[0002] In recent years, when photographing subjects such as fireworks, long-exposure photography functions such as bulb photography have made it possible to capture the entire light trail of the fireworks from the time it is launched until it disappears, thereby enabling the creation of picturesque photographs.

[0003] However, with imaging devices using CCD or CMOS sensors, the photographer cannot check the brightness (luminance value) of the subject during exposure, so they only know whether the exposure time was appropriate once they have finished shooting.In other words, because the photographer cannot determine the exposure time (timing to end shooting) according to the brightness of the subject during shooting, there is a risk of the subject being underexposed or, conversely, producing a failed image with the subject blown out.

[0004] Therefore, Patent Document 1 proposes a technique in which signals from a portion of pixels are read out sequentially during an exposure period and displayed on a display unit, and when the exposure is completed, the pixel signals are added together to generate a final image.

[0005] Furthermore, Patent Document 2 proposes a technique in which image data is read out during an exposure period, and is sequentially added, and the image data generated after the addition process is displayed on a display unit. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-05173 [Patent Document 2] Japanese Patent Application Laid-Open No. 2009-130470 [Overview of the Initiative] [Problem to be solved by the invention]

[0007] However, the technology described in Patent Document 1 increases the circuit size because the images read out during exposure are stored in multiple different memory areas. Also, since the recorded image is generated by adding two frames of images, the readout noise is added twice, which degrades the noise performance of the recorded image.

[0008] In the technology described in Patent Document 2, since the read image signals are sequentially added together, noise components are superimposed for each added frame, resulting in a decrease in the noise performance of the recorded image. Furthermore, processing such as A / D conversion and frame addition is required for each image signal read out for display, which leads to increased control complexity and longer delays until the image is displayed.

[0009] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an imaging device that is capable of acquiring subject brightness information during exposure while suppressing deterioration in image quality of recorded images. [Means for solving the problem]

[0010] The imaging device according to the present invention is characterized by comprising: an image sensor having a plurality of pixels arranged two-dimensionally, each comprising a photoelectric conversion unit that emits pulses at a frequency corresponding to the frequency of photon reception, a counter that counts the number of pulses, and a memory that stores the count value of the counter; an image processing means that generates an image signal based on the count value of the counter at the end of the exposure period; a display control means that displays information using the image signal on a display means; and a control means that causes the image sensor to perform exposure for a first exposure period and exposure for a second exposure period that includes the first exposure period and is longer than the first exposure period in a single shooting, and to display information using the first image signal generated during the first exposure period on the display means before the end of the second exposure period. [Effects of the Invention]

[0011] According to the present invention, it is possible to acquire subject brightness information during exposure while suppressing degradation of the image quality of the recorded image. [Brief explanation of the drawings]

[0012] [Figure 1] A diagram showing an example configuration of an image sensor related to one embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of the configuration of a sensor substrate. [Figure 3] FIG. 2 is a diagram showing an example of the configuration of a circuit board. [Figure 4] A diagram showing an example of an equivalent circuit for a pixel and a corresponding signal processing circuit. [Figure 5] A schematic diagram illustrating the relationship between APD operation and output signals. [Figure 6] FIG. 1 is a block diagram showing the configuration of an imaging apparatus. [Figure 7] A timing chart showing the timing of operations in system control. [Figure 8A] This figure shows an example of subject brightness information to be displayed on the display unit. [Figure 8B] This figure shows an example of subject brightness information to be displayed on the display unit. [Figure 8C] This figure shows an example of subject brightness information to be displayed on the display unit. [Figure 8D] This figure shows an example of subject brightness information to be displayed on the display unit. [Figure 8E] This figure shows an example of subject brightness information to be displayed on the display unit. [Figure 9] A flowchart illustrating the operation of the imaging device. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0014] Figure 1 shows an example of the configuration of an image sensor according to one embodiment of the present invention. In the following description, the image sensor 100 will be described as an image sensor having a so-called stacked structure, in which two substrates, a sensor substrate 11 and a circuit substrate 21, are stacked and electrically connected. However, it is not limited to this, and it may also be a so-called non-stacked structure in which the components included in the sensor substrate and the components included in the circuit substrate are arranged on a common semiconductor layer. The sensor substrate 11 includes a pixel region 12, and the circuit substrate 21 includes a circuit region 22 that processes signals detected in the pixel region 12.

[0015] 2 is a diagram showing an example of the configuration of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and columns. Each pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, referred to as APD).

[0016] Here, the photoelectric conversion unit 102 functions as a sensor unit that emits pulses at a frequency corresponding to the frequency of receiving photons. Although an example of a pixel array with 3 rows and 4 columns is shown in Fig. 2, the number of rows and columns of the pixel array that constitutes the pixel region 12 is not particularly limited.

[0017] 3 is a diagram showing an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in FIG. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, column output lines 113, a vertical scanning circuit 110, and an output circuit 114.

[0018] The vertical scanning circuit 110 receives control pulses supplied from the control pulse generation unit 115 and sequentially supplies control pulses to multiple pixels arranged in the row direction, row by row. Logic circuits such as a shift register and an address decoder are used in the vertical scanning circuit 110.

[0019] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, memory, etc., and digital values ​​are stored in the memory. The horizontal scanning circuit 111 supplies a control pulse to the signal processing circuit 103 to sequentially select each column in order to read the signal from the memory of each pixel where the digital signal is stored.

[0020] Signals are output to the column output line 113 from the signal processing circuit 103 of the pixels in the row selected by the vertical scanning circuit 110. The signals output to the column output line 113 are then output to the outside of the image sensor 100 via the readout circuit 112 and the output circuit 114.

[0021] The readout circuit 112 incorporates multiple buffers connected to each column output line 113. Furthermore, the control pulse generation unit 115 can control the vertical scanning circuit 110 to perform non-additive readout, selecting all rows.

[0022] Furthermore, to reduce the amount of data in the signal to be read, the system can be controlled to perform decimation, selecting rows to read every few rows. That is, as will be described later, the operation to read the display image may be performed in the middle of the exposure period for generating the recording image, and the amount of data may be reduced by decimating pixels in rows or columns when generating the display image.

[0023] Alternatively, it is possible to control the device to perform averaging readout, in which a plurality of adjacent pixel signals are subjected to processing such as averaging and read out as a single pixel signal. That is, when generating an image for display as described above, the amount of data may be reduced by adding the count values ​​of a plurality of pixels. In this case, the number of pixels to be averaged may be freely changed in both the row and column directions, such as 2x2 pixels or 4x4 pixels.

[0024] Alternatively, in order to reduce the amount of data of the readout signals, control may be performed to perform crop readout, which selects only a partial pixel area (angle of view). That is, when generating an image for display as described above, the amount of data may be reduced by cropping pixels in a partial area and reading out the signals.

[0025] By performing thinning readout, averaging readout, and crop readout to reduce the amount of data, it becomes possible to read out image signals at high speed, thereby shortening the delay time until the image is displayed.

[0026] 2 and 3, a plurality of signal processing circuits 103 are arranged in an area that overlaps vertically with the pixel area 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in an area that overlaps vertically with a peripheral area (non-pixel area) of the pixel area 12.

[0027] The arrangement of the column output lines 113, the readout circuits 112, and the output circuits 114 is not limited to the example shown in Fig. 3. For example, the column output lines 113 may be arranged extending in the row direction, and the readout circuits 112 may be arranged at the ends of the column output lines 113. Furthermore, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit, and one signal processing circuit may be shared by multiple photoelectric conversion units to perform signal processing sequentially.

[0028] FIG. 4 is a diagram showing an equivalent circuit of the pixel 101 in FIGS. 2 and 3 and the signal processing circuit 103 corresponding to the pixel 101. In FIG.

[0029] The APD 201 included in the photoelectric conversion unit 102 generates charge pairs according to incident light through photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line that supplies a drive voltage VL (first voltage). The other of the two nodes of the APD 201 is connected to a power supply line that supplies a drive voltage VH (second voltage) that is higher than the drive voltage VL.

[0030] In Figure 4, one node of the APD 201 is the anode, and the other node of the APD 201 is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. With this voltage supplied, the charge generated by incident light undergoes avalanche multiplication, generating an avalanche current.

[0031] When a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the voltage difference between the anode and cathode is close to or less than the breakdown voltage. An APD operating in Geiger mode is called a SPAD. In the case of a SPAD, for example, the drive voltage VL (first voltage) is -30V and the drive voltage VH (second voltage) is 1V.

[0032] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quench element 202 is connected to a power line to which a drive voltage VH is supplied, and to one of the nodes, either the anode or the cathode, of the APD 201.

[0033] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to the APD 201 and suppressing avalanche multiplication (quench operation).The quench element 202 also functions to return the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0034] In the configuration shown in FIG. 4, the signal processing circuit 103 includes a waveform shaping section 210, a counter circuit 211, and a memory circuit 212 in addition to the quench element 202.

[0035] The waveform shaping unit 210 shapes the voltage change at the cathode of the APD 201 obtained when photons are detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. While Fig. 4 shows an example in which one inverter is used as the waveform shaping unit 210, a circuit in which multiple inverters are connected in series, or another circuit with a waveform shaping effect, may also be used.

[0036] Counter circuit 211 counts the number of pulses output from waveform shaping unit 210 and holds the count value. When control pulse RESET is supplied via drive line 213, the signal held in counter circuit 211 is reset. Here, counter circuit 211 generates a signal based on the difference between the count values ​​at the start and end of the accumulation period.

[0037] A control pulse SEL is supplied to the memory circuit 212 from the vertical scanning circuit 110 in FIG. 3 via a drive line 214 in FIG. 4 (not shown in FIG. 3), and the counter circuit 211 and the column output line 113 are electrically connected or disconnected from each other.

[0038] The memory circuit 212 functions as a memory that temporarily stores the counter's count value and outputs the pixel output signal from the counter circuit 211 to the column output line 113. The memory circuit 212 may also be configured to be electrically connected to the signal processing circuit 103 shown in Figure 4 and to a counter circuit located within the adjacent signal processing circuit 103 (not shown in Figure 4) to perform operations such as averaging.

[0039] By doing so, multiple pixel signals generated by multiple photoelectric conversion units can be averaged or otherwise combined to output a single pixel signal, thereby reducing the amount of data in the pixel output signal.

[0040] Alternatively, the memory circuit 212 may be configured to selectively switch between a non-additive read mode, which is connected to only one counter circuit 211, and an averaging read mode, which is connected to multiple counter circuits.

[0041] Furthermore, switches such as transistors may be placed between the quench element 202 and the APD201, or between the photoelectric conversion unit 102 and the signal processing circuit 103, to switch the electrical connections. Similarly, the supply of the drive voltage VH or drive voltage VL to the photoelectric conversion unit 102 may be electrically switched using switches such as transistors.

[0042] Figure 5 schematically shows the relationship between the operation of APD201 and the output signal. The input side of the waveform shaping unit 210 is nodeA, and the output side is nodeB. Between time t50 and time t51, a potential difference of VH-VL is applied to APD201. When a photon is incident on APD201 at time t51, avalanche multiplication occurs in APD201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0043] As the voltage drop increases further and the potential difference applied to APD201 decreases, the avalanche multiplication of APD201 stops, as shown at time t52, and the voltage level of nodeA no longer drops below a certain value.

[0044] After that, between time t52 and time t53, a current flows through node A to compensate for the voltage drop from voltage VL, and at time t53, node A settles to its original potential level. At this time, the portion of the output waveform at node A that exceeds a certain threshold is shaped by waveform shaping unit 210 and output as a pulse signal at node B.

[0045] Next, the configuration of the imaging device 600 in this embodiment will be described. Figure 6(a) is a block diagram of the imaging device 600 in this embodiment.

[0046] The imaging device 600 includes the imaging element 100 described in FIGS. 1 to 5, an optical system 601, a signal processing unit 602, a memory 603, a display unit (display control unit) 604, a recording unit 605, an operation unit 606, a control unit 607, and the like.

[0047] The image sensor 100 is configured by a two-dimensional array of pixels, each having an avalanche photodiode for photoelectrically converting an optical image, as described with reference to Figures 1 to 5. Based on a control signal from the control unit 607, the image sensor 100 can selectively switch between a number of readout modes, such as a non-additive readout mode, an averaging readout mode, a thinning readout mode, and a crop readout mode.

[0048] The non-additive readout mode is a mode in which signals from all pixels are read as they are, the arithmetic average readout mode is a mode in which signals from adjacent pixels are averaged and read out, the thinning readout mode is a mode in which pixels are thinned out and read out, and the crop readout mode is a mode in which signals from only a partial pixel area (field of view) are read out.

[0049] In this embodiment, the display image to be displayed on the display unit 604 is read out in a 2x2 pixel additive average readout mode, while the recording image to be recorded on the recording unit 605 is read out in a non-additive readout mode. That is, since the amount of data read out for the display image is 1 / 4 that of the recording image, the time required for reading is also 1 / 4. In other words, the pixel signals for the display image can be read out four times faster than for the recording image, so the delay time of the display image relative to the actual state of the subject can be shortened compared to the non-additive readout mode.

[0050] The signal processing unit 602 performs processing on the image signal acquired by the image sensor 100 to generate subject brightness information. Specifically, subject brightness information includes image information that directly shows the brightness values ​​of the subject as an image, histogram information that shows the frequency of occurrence of each brightness level in the image, and blown-out pixel information that shows information about pixels where the brightness value is blown out. In addition, information generated by combining multiple pieces of this information is also considered subject brightness information. For example, image information with histogram information superimposed on it, or image information after processing in which only the blown-out pixels in the image information are replaced with a zebra pattern or a fixed color, is also considered subject brightness information.

[0051] Figure 6(b) is a functional block diagram showing an example of the configuration of the signal processing unit 602. In Figure 6(b), the signal processing unit 602 is configured to include an image processing unit 6021, a histogram generation unit 6022, a highlight detection unit 6023, and a brightness information generation unit 6024. In this embodiment, three types of processing units are provided to generate subject brightness information, but the means for generating subject brightness information are not limited to these three processing units.

[0052] The image processing unit 6021 performs image processing on the input image signal. For example, image processing such as light emission crosstalk correction, noise reduction, gamma correction, white balance correction, demosaic, and data compression is performed. After performing these image processing operations, the image information is output to the luminance information generation unit 6024. In this embodiment, image processing is performed on both the display image and the recording image, but the image processing method may be different for display and recording. For example, for the display image, processing unnecessary for checking the subject luminance may be omitted and the amount of data may be thinned out to speed up processing in order to display the image with as little delay as possible, and image processing may be performed on the recording image to achieve the highest possible image quality.

[0053] The histogram generation unit 6022 generates distribution information of brightness values ​​from the input image signal. The frequency of occurrence of each brightness value is counted and graphed as histogram information. The generated histogram information is output to the brightness information generation unit 6024.

[0054] The blown-out highlight detection unit 6023 detects blown-out highlight pixels from the input image signal. In this embodiment, a threshold value for determining blown-out highlights is determined in advance, and pixel addresses exceeding the threshold value are listed and generated as blown-out highlight pixel information. However, this is not limited to this as long as the method for determining blown-out highlight pixel information can be used. The generated blown-out highlight pixel information is output to the luminance information generation unit 6024.

[0055] The luminance information generation unit 6024 selects information to be output as subject luminance information from the image information output from the image processing unit 6021, the histogram information output from the histogram generation unit 6022, and the blown-out highlight pixel information output from the blown-out highlight detection unit 6023. It also has a function to combine multiple pieces of luminance information when it is desired to output these pieces of information. Histogram information may be combined with the image information output from the image processing unit 6021, or blown-out highlight pixels in the image signal output from the image processing unit 6021 may be replaced with other values ​​based on the blown-out highlight pixel information. The subject luminance information to be output from the luminance information generation unit 6024 may be determined in advance, or may be selectable based on an operational instruction from the photographer.

[0056] The signal processing unit 602 outputs the subject luminance information (one or a combination of image information, histogram information, and blown-out highlight pixel information) generated by the luminance information generation unit 6024 to the display unit 604 and the recording unit 605. In this embodiment, the subject luminance information displayed on the display unit 604 is information obtained by combining image information and histogram information, and the subject luminance information to be recorded in the recording unit 605 is image information only. Furthermore, at the timing when the image information is recorded in the recording unit 605, the display unit 604 simultaneously displays information obtained by combining image information and histogram information.

[0057] Memory 603 is a means for temporarily storing data when the signal processing unit 602 performs various processing. For example, it is used to temporarily store image data for various image processing in the image processing unit 6021, or for calculations such as counting histograms and blown-out pixels.

[0058] The display unit 604 includes a display element such as a liquid crystal device or an organic EL element, and displays the subject brightness information output from the signal processing unit 602 .

[0059] The recording unit 605 includes, for example, a recording medium such as a memory card or a hard disk, and records and stores image information. If necessary, it may also record subject brightness information in addition to image information.

[0060] The control unit 606 is used by the photographer to perform a shutter release operation to determine the start and end timing of exposure, and to select what subject brightness information to output to the display unit 604.

[0061] The control unit 607 incorporates a CPU as a computer and memory that stores computer programs, and controls each part of the imaging device 600 by executing the computer programs stored in the memory.

[0062] For example, the control pulse generation unit 115 of the image sensor 100 controls the length of the exposure period (photoelectric conversion period) for each frame and subframe of the image sensor 100, as well as the timing of the control signal CLK. In addition, multiple readout methods such as non-additive readout mode, averaging readout mode, decimation readout mode, and crop readout mode are selectively switched based on the control specifications. In this embodiment, the averaging readout mode is selected when acquiring an image for display, and the non-additive readout mode is selected when acquiring an image for recording.

[0063] Furthermore, for example, the image processing unit 6021 within the signal processing unit 602 sets parameters for image processing, generates histogram information using the histogram generation unit 6022, and controls the generation of overexposure pixel information using the overexposure detection unit 6023. Then, the luminance information generation unit 6024 selects and combines this subject luminance information and controls it to output to the display unit 604 and the recording unit 605.

[0064] Furthermore, it has a function to receive operation instructions from the photographer via the control unit 606. Specifically, it accepts operation instructions such as shutter release and selection of the type of subject brightness information to be displayed on the display unit.

[0065] Although the configuration of this embodiment has been described above, the present invention is not limited to the above configuration.

[0066] Figure 7 is a timing chart showing the timing of system control in this embodiment, illustrating the processing from when the photographer performs the shutter release operation until the image information is recorded in the recording unit 605.

[0067] In this embodiment, the control unit 607 acquires a total of four image signals from the image sensor 100 in one shooting session: three subframes (subframes 1 to 3) and one main frame. Of the four frames, subframes 1 to 3 are images to be output to the display unit 604, and image signals are read out from the image sensor 100 in an additive average readout mode. The main frame is an image to be recorded in the recording unit 605, and is an image read out from the image sensor 100 in a non-additive readout mode. The image signal for the display image can be read out in one-fourth the time required for the image for recording. Furthermore, in this embodiment, the image information is also displayed on the display unit 604 together with histogram information so that the photographer can confirm the image information finally recorded in the recording unit 605.

[0068] First, at time t700, the photographer uses the control unit 606 to perform a shutter release. The shutter release switches the output signal of the control unit 606 from Lo to Hi, and exposure takes place while the Hi state is maintained. The shutter release operation is held even after time t701.

[0069] At time t701, a reset process is performed on the image sensor 100. Specifically, the control unit 607 supplies a control pulse RESET to the counter circuit 211 via the drive line 213, thereby resetting the count value held by the counter circuit 211. During the period from time t701 to time t702, the control pulse RESET remains Hi, and at the timing of time t702, the control pulse RESET becomes Lo, releasing the reset of the counter circuit 211.

[0070] At time t702, exposure to pixel 101 begins and continues until the shutter release is canceled.

[0071] At time t703, output of the image signal for subframe 1 begins. Specifically, processing begins to write the count value of the counter circuit 211 to the memory circuit 212. At time t704, when the processing to write to the memory circuit 212 is completed, the count values ​​for subframe 1 are stored in the memory circuits 212 of all pixels.

[0072] At time t704, the count values ​​for subframe 1 stored in memory circuit 212 are sequentially output from image sensor 100 to signal processing unit 602. Then, upon receiving the image signal (count value) for subframe 1, signal processing unit 602 begins generating subject luminance information to be displayed on display unit 604.

[0073] In this embodiment, a process for generating image information using an image processing unit 6021 and a process for generating histogram information using a histogram generation unit 6022 are performed in parallel. Once the image information and histogram information have been generated, final subject luminance information is generated using a luminance information generation unit 6024. Specifically, a synthesis process is performed in which the histogram information is superimposed on the image information, and a synthesized image such as that shown in FIG. 8A is generated at time t705.

[0074] In subframe 1, the fireworks have just been launched, so the image signal is dark, and the histogram information is dominated by mostly low-brightness pixels (the dashed line in the histogram). However, it can be seen that only the trajectory of the launched fireworks contains pixels with slightly higher brightness (the solid line in the histogram). The generated composite image is temporarily stored in memory 603 as subject brightness information to be output to display unit 604.

[0075] At time t705, the subject luminance information stored in memory 603 is output to display unit 604, and display begins. The image signal of subframe 1 continues to be displayed on display unit 604 until the next frame (subframe 2) is acquired.

[0076] Next, the same processes as those for obtaining an image signal and generating subject luminance information for subframe 1 are performed for subframes 2 and 3. Specifically, to generate subject luminance information for subframe 2, the same processes as those performed from time t703 to time t705 are performed from time t706 to time t708. The composite image generated at time t708 is shown in FIG. 8B. In subframe 2, the fireworks are just beginning to explode, so there is an increase in pixels with relatively high luminance compared to subframe 1.

[0077] Once generation of the object luminance information for subframe 2 has finished, a predetermined time elapses and then object luminance information for subframe 3 is generated. To generate object luminance information for subframe 3, the same processing as from time t703 to time t705 is performed from time t709 to time t711. The composite image generated at time t711 is shown in FIG. 8C.

[0078] In subframe 3, the fireworks are in the middle of opening, so there are more high-brightness pixels than in subframe 2, and pixels whose count values ​​are close to saturation are beginning to appear. The photographer can freely determine the timing of releasing the release while checking the changes in the subject's brightness value in Figures 8A to 8C, which show the subject brightness information for subframes 1 to 3. In Figure 8C, taking into consideration that pixels whose count values ​​are close to saturation are beginning to appear and that the fireworks have opened to a certain extent, the photographer releases the release at time t712. Releasing the release switches the output signal of the operation unit 606 from Hi to Lo.

[0079] At time t712, output of the main frame image signal begins. As with times t703, t706, and t709, the process involves writing the count value of the counter circuit 211 to the memory circuit 212. Because the main frame is in non-additive readout mode, the count value is written to the memory circuit 212 for each pixel without addition. At time t713, when the write process to the memory circuit 212 is completed, the count values ​​of all pixels in the main frame are stored in the memory circuit 212.

[0080] At time t713, the count values ​​of the main frames stored in the memory circuit 212 are sequentially output from the image sensor 100 to the signal processing unit 602. Then, the signal processing unit 602, which has received the image signals of the main frames, starts generating image information to be recorded in the recording unit 605. In addition, processing for generating subject brightness information to be displayed on the display unit 604 is also performed in parallel.

[0081] Since the main frame is in non-additive readout mode, it takes four times as long to process as subframes 1 to 3, which are subjected to additive average readout. Also, since the purpose is to record images, image processing unit 6021 is used to perform processing that prioritizes image quality.

[0082] FIG. 8D shows image information to be recorded in the recording unit 605, and FIG. 8E shows subject brightness information to be displayed on the display unit 604.

[0083] Fig. 8D shows the image information itself processed by the image processing unit 6021, as it is intended to be recorded in the recording unit 605. On the other hand, Fig. 8E shows an image for allowing the photographer to visually confirm the finally recorded image using the display unit 604. Although it is not essential to combine histogram information with the image information, in this embodiment, the histogram information is also superimposed so that it can be easily confirmed whether the finally recorded image has been subjected to whiteout.

[0084] 8D and 8E, it can be seen that the fireworks have fully opened and the exposure was completed just before the brightness value reached saturation. The image for recording shown in Fig. 8D and the image for display shown in Fig. 8E are temporarily stored in memory 603.

[0085] At time t714, the signal processing unit 602 outputs the image information and subject brightness information held in the memory 603 to the display unit 604 and the recording unit 605. Specifically, it outputs the recording image shown in Figure 8D to the recording unit 605, and the recording unit 605 records the acquired image information to the recording medium. It also outputs the subject brightness information for display, shown in Figure 8E, to the display unit 604, and the display unit 604 displays the acquired subject brightness information on the display. The photographer can then make a final determination as to whether the image was taken with the appropriate exposure time by checking the subject brightness information displayed on the display unit 604.

[0086] Figure 9 is a flowchart showing the operation of the system control in this embodiment. Using Figure 9, and referring to the timing chart in Figure 7, the operation of the imaging device will be explained.

[0087] In S901, the control unit 607 determines whether the photographer has performed the shutter release operation (~time t700). If the control unit 607 determines that the photographer has performed the shutter release operation, it proceeds to S902; otherwise, it repeats the process in S901.

[0088] In S902, the control unit 607 resets the count value of the counter circuit 211 (time t701 to time t702). Once the count value reset is complete, the control unit 607 proceeds to S903.

[0089] In S903, the control unit 607 causes the image sensor 100 to start exposure (time t702). After starting exposure, the control unit 607 advances the process to S904.

[0090] In S904, the control unit 607 determines whether or not exposure is continuing, that is, whether or not the photographer has activated the shutter release operation (whether or not it has been deactivated). If it is determined that exposure is continuing (times t702 to t712), the control unit 607 proceeds to S905. If it is determined that exposure is not continuing (times t712 onwards), the control unit 607 proceeds to S910.

[0091] In S905, the control unit 607 determines whether or not it is time to acquire a display image. If it is time to acquire a display image (time t703, time t706, time t709), the control unit 607 proceeds to S906. If it is not time to acquire a display image, the process in S905 is repeated.

[0092] In S906, the control unit 607 acquires the display image. Once the acquisition of the display image is complete (times t704, t707, and t710), the control unit 607 proceeds to process S907.

[0093] In S907, the control unit 607 generates subject brightness information from the display image acquired from the image sensor 100 (times t704-t705, t707-t708, t710-t711). Specifically, it generates information that reveals the subject brightness by combining the aforementioned image signal, histogram information, and blown-out pixel information (times t705, t708, t711). Once the subject brightness information has been generated, the control unit 607 proceeds to processing in S908.

[0094] In S908, the control unit 607 starts displaying subject brightness information using the display unit 604. Once the control unit 607 starts displaying subject brightness information (at times t705, t708, and t711), it proceeds to S909.

[0095] In S909, the control unit 607 determines the timing for acquiring the next display image. The display image may be acquired at a predetermined interval, or the timing for acquiring the next display image may be determined based on the subject brightness information generated in S907. For example, based on the subject brightness information, if there are few pixels that are likely to be overexposed, the time until acquiring the next image should be increased, and if there are many pixels that are likely to be overexposed, the time until acquiring the next display image should be decreased. It is also advisable to store these timings in advance as a program. Once the control unit 607 has determined the timing for acquiring the next display image, it returns to processing in S904.

[0096] On the other hand, in S910, the control unit 607 acquires an image for recording (time t712 to time t713). When the control unit 607 completes acquiring the image for recording (time t713), the process proceeds to S911.

[0097] In S911, the control unit 607 generates a final image for recording from the image signal acquired from the image sensor 100 (time t713 to time t714). Once the control unit 607 has generated the subject brightness information, the process proceeds to S912.

[0098] In S912, the control unit 607 records (from time t714) the image for recording in the recording unit 605. When the recording of the image for recording is completed, this flow ends.

[0099] As described above, in this embodiment, apart from the image recorded in the recording unit 605, subject brightness information is generated from an image read during exposure and displayed on the display unit 604, allowing the photographer to freely determine the timing to end exposure.

[0100] In this embodiment, the histogram information is generated based on the assumption that a histogram is generated for the entire area of ​​the image signal, but it is also possible to use luminance information for only a specific area, such as fireworks. This prevents the photographer from accidentally ending exposure based on information other than the main subject.

[0101] In this embodiment, subject brightness information is displayed on the display unit 604 so that the photographer can end exposure at a timing of their choice, but the control unit 607 may also end exposure automatically based on a program. For example, the number of blown-out highlight pixels is counted using the blown-out highlight detection unit 6023, and if the count value reaches a predetermined value or more, processing is performed to automatically end exposure. This makes it possible to end exposure at an appropriate timing without imposing a burden on the photographer.

[0102] (Other embodiments) Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0103] The disclosures herein include the following imaging devices and their control methods, programs, and storage media.

[0104] (Item 1) an image sensor in which a plurality of pixels are two-dimensionally arranged, each of which includes a photoelectric conversion unit that emits pulses at a frequency corresponding to the frequency of receiving photons, a counter that counts the number of said pulses, and a memory that stores the count value of said counter; Image processing means that generates an image signal based on the count value of the counter at the end of the exposure period, A display control means for displaying information using the aforementioned image signal, Control means for causing the image sensor to perform exposure for a first exposure period and exposure for a second exposure period that includes the first exposure period but is longer than the first exposure period, and for causing the display control means to display information using the first image signal generated during the first exposure period on the display means until the end of the second exposure period, An imaging device characterized by comprising:

[0105] (Item 2) The imaging apparatus according to item 1, characterized in that the information using the first image signal includes image information generated based on the first image signal.

[0106] (Item 3) 3. The imaging device according to item 1 or 2, wherein the information using the first image signal includes information about the brightness of the subject generated using the first image signal.

[0107] (Item 4) 4. The imaging device according to item 3, wherein the information regarding the brightness of the subject includes a histogram of brightness of the subject for each pixel, generated using the first image signal.

[0108] (Item 5) 4. The imaging device according to item 3, wherein the information about the brightness of the subject includes information about blown-out white pixels generated using the first image signal.

[0109] (Item 6) 6. The imaging device according to item 5, wherein the control means terminates exposure of the imaging element when the number of blown-out pixels in the blown-out pixel information becomes equal to or greater than a predetermined number.

[0110] (Item 7) 7. The imaging device according to any one of items 3 to 6, wherein the control means generates information about the brightness of the subject using signals of pixels in a predetermined region in the first image signal.

[0111] (Item 8) The imaging device described in any one of items 1 to 7, characterized in that the control means generates the first image signal using pixel signals of a number of pixels that is reduced from the total number of pixels of the imaging element.

[0112] (Item 9) The imaging device described in item 8 is characterized in that the control means generates the first image signal using pixel signals of a number of pixels that is reduced from the total number of pixels of the imaging element by performing pixel addition processing, pixel thinning processing, and area clipping processing.

[0113] (Item 10) 10. The imaging device according to item 8 or 9, wherein the control means generates the second image signal generated during the second exposure period using signals from all pixels of the imaging element.

[0114] (Item 11) 11. The imaging device according to any one of items 1 to 10, wherein the control means determines the timing for acquiring the next image based on information using the first image signal.

[0115] (Item 12) 12. The imaging device according to any one of items 1 to 11, wherein the control means controls the image sensor to perform exposure for a third exposure period that includes the first exposure period and is longer than the first exposure period and shorter than the second exposure period in one shooting, and controls the display control means to display information using a third image signal generated during the third exposure period on the display means between the end of the first exposure period and the end of the second exposure period.

[0116] (Item 13) The imaging device described in any one of items 1 to 12, characterized in that the imaging element is an imaging element having a stacked structure configured by stacking a first substrate on which the photoelectric conversion unit is arranged and a second substrate on which the counter and the memory are arranged.

[0117] (Item 14) A method for controlling an imaging device having an imaging element in which a plurality of pixels are two-dimensionally arranged, each of which includes a photoelectric conversion unit that emits pulses at a frequency corresponding to a frequency of receiving photons, a counter that counts the number of the pulses, and a memory that stores the count value of the counter, An image processing step that generates an image signal based on the count value of the counter at the end of the exposure period, A display control step that causes information using the aforementioned image signal to be displayed on a display means, a control step of causing the image sensor to perform exposure for a first exposure period and exposure for a second exposure period that includes the first exposure period and is longer than the first exposure period in one shooting, and controlling the display control step so that information using a first image signal generated in the first exposure period is displayed on the display means until the end of the second exposure period; A control method for an imaging device, characterized by having the following features.

[0118] (Item 15) 15. A program for causing a computer to execute each step of the method for controlling an imaging device according to item 14.

[0119] (Item 16) Item 15. A computer-readable storage medium storing a program for causing a computer to execute each step of the method for controlling an imaging device according to Item 14.

[0120] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0121] 600: imaging device, 601: optical system, 602: signal processing unit, 603: memory, 604: display unit, 605: recording unit, 606: operation unit, 607: control unit, 6021: image processing unit, 6022: histogram generation unit, 6023: overexposure detection unit, 6024: luminance information generation unit

Claims

1. an image sensor in which a plurality of pixels are two-dimensionally arranged, each of which includes a photoelectric conversion unit that emits pulses at a frequency corresponding to the frequency of receiving photons, a counter that counts the number of the pulses, and a memory that stores the count value of the counter; an image processing means for generating an image signal based on the count value of the counter at the end of the exposure period; a display control means for displaying information using the image signal on a display means; a control means for controlling the display control means to cause the image sensor to perform exposure for a first exposure period and exposure for a second exposure period that includes the first exposure period and is longer than the first exposure period in one photographing operation, and to cause the display means to display information using a first image signal generated in the first exposure period until the end of the second exposure period; An imaging device comprising:

2. 2. The imaging device according to claim 1, wherein the information using the first image signal includes image information generated based on the first image signal.

3. 2. The imaging device according to claim 1, wherein the information using the first image signal includes information about the luminance of the subject generated using the first image signal.

4. 4. The imaging device according to claim 3, wherein the information about the luminance of the object includes a histogram of the luminance of the object for each pixel, which is generated using the first image signal.

5. 4. The imaging device according to claim 3, wherein the information about the luminance of the subject includes information about blown-out white pixels generated using the first image signal.

6. 6. The imaging apparatus according to claim 5, wherein the control means terminates exposure of the imaging element when the number of blown-out highlight pixels in the blown-out highlight pixel information reaches a predetermined number or more.

7. 4. The imaging apparatus according to claim 3, wherein the control means generates information relating to the luminance of the subject by using signals of pixels in a predetermined region in the first image signal.

8. 2. The imaging device according to claim 1, wherein the control means generates the first image signal using pixel signals of a number of pixels that is reduced from the total number of pixels of the imaging element.

9. The imaging device according to claim 8, wherein the control means generates the first image signal using pixel signals of a number of pixels that is reduced from the total number of pixels of the imaging element by performing pixel addition processing, pixel thinning processing, and area cutout processing.

10. 9. The imaging device according to claim 8, wherein the control means generates the second image signal generated during the second exposure period using signals from all pixels of the imaging element.

11. 2. The imaging device according to claim 1, wherein the control means determines the timing for acquiring the next image based on information obtained using the first image signal.

12. 2. The imaging device according to claim 1, wherein the control means controls the image sensor to perform exposure for a third exposure period that includes the first exposure period and is longer than the first exposure period but shorter than the second exposure period in one shooting operation, and controls the display control means to display information using a third image signal generated during the third exposure period on the display means between the end of the first exposure period and the end of the second exposure period.

13. 2. The imaging device according to claim 1, wherein the imaging element is an imaging element having a stacked structure in which a first substrate on which the photoelectric conversion unit is arranged and a second substrate on which the counter and the memory are arranged are stacked.

14. A method for controlling an imaging device including an imaging element in which a plurality of pixels are two-dimensionally arranged, each of which includes a photoelectric conversion unit that emits pulses at a frequency corresponding to a frequency of receiving photons, a counter that counts the number of the pulses, and a memory that stores the count value of the counter, comprising: an image processing step of generating an image signal based on the count value of the counter at the end of the exposure period; a display control step of displaying information using the image signal on a display means; a control step of causing the image sensor to perform exposure for a first exposure period and exposure for a second exposure period that includes the first exposure period and is longer than the first exposure period in one photographing operation, and controlling the display control step so that information using a first image signal generated in the first exposure period is displayed on the display means until the end of the second exposure period; 10. A method for controlling an imaging device, comprising:

15. A program for causing a computer to execute each step of the method for controlling an imaging apparatus according to claim 14.

16. 15. A computer-readable storage medium storing a program for causing a computer to execute each step of the method for controlling an imaging apparatus according to claim 14.

Images