Imaging apparatus, control method, program, and storage medium
By counting the number of avalanche discharges and adjusting the exposure parameters, the exposure instability problem caused by the dark current changes of the SPAD sensor is solved, and stable automatic exposure control is achieved.
Patent Information
- Application Number
- JP2023122732
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-07-27
- Publication Date
- 2025-02-06
AI Technical Summary
The existing SPAD sensors have dark current changes during long-term use, resulting in inaccurate brightness capture, and automatic exposure adjustment cannot achieve the best effect, resulting in unstable exposure.
By counting the number of avalanche discharges, an exposure control signal is generated using the generator, calculation unit and storage unit to adjust the exposure parameters to stabilize the exposure effect.
It realizes stable automatic exposure adjustment when the characteristics of SPAD sensors change, ensuring the accuracy and stability of image brightness capture.
Smart Images
Figure 2025018749000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an imaging device, a control method, a program, and a storage medium. [Background technology]
[0002] One type of image sensor is the Single Photon Avalanche Diode (SPAD) sensor (hereafter referred to as the SPAD sensor). The SPAD sensor uses the avalanche amplification phenomenon, in which electrons are accelerated by applying a strong electric field, and then collide with other electrons to provoke multiple electrons, creating an avalanche-like phenomenon that generates a large current. This allows weak photons that enter a pixel to be converted into a large current that can be detected as an electric charge. Because of the mechanism behind the SPAD sensor, no noise is introduced during signal readout, so it is expected to be used as an image sensor. In particular, because it can clearly capture subjects without being affected by noise even in dark places, it is expected to be widely used as an image sensor for surveillance applications, etc.
[0003] When considering surveillance applications, since the camera will be set up in a fixed location and will continue to capture images for long periods of time, it is desirable for the exposure to be automatically adjusted according to the brightness of the subject. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] C. Zhang, “SPAD requirements from consumer electronics to automotive”, Int. SPAD Sensor Workshop(2022) Summary of the Invention [Problem to be solved by the invention]
[0005] Non-Patent Document 1 reports a phenomenon in which the amount of dark current changes when a photoelectric conversion device having an APD (Avalanche Photo Diode) is driven for a long time. In addition, since the dark current may change or increase depending on the number of avalanches, the degree to which the dark current changes or increases may differ depending on the region.
[0006] Since the dark current changes for each region depending on the number of avalanches, the brightness of the subject cannot be captured correctly depending on the level of the dark current. When performing automatic exposure adjustment for surveillance purposes, this can lead to unstable phenomena such as the automatic exposure adjustment not reaching the optimal brightness or passing the optimal brightness multiple times and the automatic exposure not being able to stop.
[0007] The present invention has been made in view of the above problems, and has an object to provide a stable automatic exposure adjustment function even when the characteristics of the SPAD sensor change due to avalanche amplification. [Means for solving the problem]
[0008] In order to achieve the above-mentioned object, an imaging device according to one aspect of the present invention has an imaging element constituted by an avalanche photodiode for photoelectrically converting an optical image, a generation unit that generates an image signal based on an output signal from the imaging element, a calculation unit that calculates the number of occurrences of avalanche amplification in the imaging element based on the image signal, a memory unit that stores a cumulative value of the number of occurrences calculated by the calculation unit, and a control unit that controls parameters related to the exposure when it is determined that the brightness of an image based on the image signal is outside a predetermined range, and is characterized in that the control unit determines the predetermined range based on the cumulative value stored in the memory unit. Effect of the Invention
[0009] According to the present invention, exposure can be stably controlled even if the characteristics of the SPAD sensor change. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a block diagram showing the internal configuration of an imaging apparatus. [Diagram 2] FIG. 2 is an equivalent circuit diagram of a pixel that constitutes an image sensor. [Diagram 3] Image signal generation flow. [Figure 4] 1 is a graph showing the relationship between the pulse count number, the photon number, and the image signal. [Diagram 5] 4 is a flowchart of an automatic exposure control function. [Figure 6] 4A and 4B are diagrams showing examples of captured images and signal values of the captured images. [Figure 7] FIG. 11 is a diagram showing an example of a weighting table used for evaluating brightness. [Figure 8] FIG. 2 is a diagram showing an example of an AE diagram. [Figure 9] 13 is a flowchart for calculating a cumulative count value. [Figure 10] 1 is a graph showing the relationship between the number of avalanche amplifications and the amount of noise generated per unit area. [Figure 11] 11A and 11B are diagrams showing examples of changes in image brightness with exposure control using aperture or exposure time. [Figure 12] 11A and 11B are diagrams illustrating an example of changes in image brightness in exposure control of digital gain. [Figure 13] 11 is a flowchart illustrating updating of exposure parameters based on the number of avalanche amplifications by region. [Figure 14] An example of cumulative count values by area. [Figure 15] FIG. 13 is a graph showing the relationship between the cumulative count value by region and the exposure response coefficient. [Figure 16] 13 is a diagram showing the relationship between the cumulative count value by region and a threshold value for determining proper exposure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, the embodiment for carrying out the present invention will be described in detail with reference to the drawings. The embodiment described below is one example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiment.
[0012] <First embodiment> (Device configuration) 1 is a block diagram showing the configuration of an imaging device according to this embodiment. The imaging optical system 10 includes a zoom lens, a focus lens, an anti-vibration lens, an aperture, an ND (Neutral Density) filter, etc. Each component of the imaging optical system 10 is controlled to determine the angle of view, focus, zoom, exposure amount, etc. when imaging a subject. The imaging optical system 10 forms an optical image of the subject on an imaging element 11.
[0013] The image sensor 11 is an image sensor that performs photoelectric conversion to convert an optical image formed at each pixel on the imaging surface into an electrical signal. Each pixel constituting the image sensor 11 is an avalanche photodiode that performs photoelectric conversion on the optical image, and functions as a SPAD sensor that counts the number of photons that form the optical image. An output signal from the image sensor 11 is output to a signal processor 12, where various image processes are performed.
[0014] The signal processing unit 12 is an image processing unit that generates an image (image signal) based on the output signal from the imaging element 11. The signal processing unit 12 is an image processing engine that performs correction processing such as removal of fixed pattern noise included in the output signal from the imaging element 11, brightness correction by digital gain, demosaicing processing, edge enhancement processing, and gamma processing. In addition to the above correction processing, the signal processing unit 12 also performs recognition processing to detect a subject area from an image (image signal) in order to control the imaging optical system 10. Furthermore, the signal processing unit 12 also generates evaluation values for exposure control and WB correction, and the generated evaluation values are transmitted to the control calculation unit 14. Specific internal processing contents in the imaging element 11 and the signal processing unit 12 will be described in detail later. The image corrected by the signal processing unit 12 is transmitted to the video output unit 13.
[0015] The video output unit 13 outputs the corrected image to a device outside the camera connected via an output terminal. It receives the image size and video frame rate that can be received by the external device, and outputs a synchronization signal for the image together with the image to be output. The output terminal may be a terminal capable of exchanging video signals with the external device. It may be a terminal such as SDI (Serial Digital Interface), HDMI (registered trademark) (High-Definition Multimedia Interface), USB (Universal Serial Bus), or RJ45. It may also be a terminal conforming to a unique standard capable of exchanging with a specific external device.
[0016] The control and calculation unit 14 generates control information to be transmitted to the imaging optical system 10, the imaging element 11, the signal processing unit 12, and the video output unit 13. The control and calculation unit 14 includes a CPU 15, and the control information is generated by the CPU 15 executing a control program stored in the memory 16 or an auxiliary storage device (not shown).
[0017] Memory 16 is an area for storing data necessary for the calculation of control information, and data during calculation and calculation results are stored in memory 16, and calculations are performed in control calculation unit 14 while CPU 15 appropriately refers to memory 16. Also, image signals generated in signal processing unit 12 may be stored in memory 16 via control calculation unit 14. Also, although memory 16 is shown here as a block diagram inside the camera, it may be configured as a removable storage device mounted inside the camera.
[0018] (Configuration of image sensor) Next, the pixel of the SPAD sensor in the image sensor 11 will be described with reference to the equivalent circuit diagram of the pixel in FIG. 2. Each pixel is composed of a photodiode 20, a quench resistor 21, and a buffer 22, and the photodiode 20 is a SPAD. When a reverse voltage exceeding the breakdown voltage is applied to the photodiode 20, the photodiode 20 operates in a Geiger mode triggered by the input of a photon. Therefore, a large reverse bias voltage equal to or greater than the breakdown voltage of the voltage VH applied to the SPAD is required. When a photon is input to the photodiode 20, an avalanche current is generated. The quench voltage VQ applied to the quench resistor 21 is configured to be variable, and the reverse bias voltage is lowered by the quench resistor value with respect to the avalanche current generated in the photodiode 20, making it possible to control the end of the avalanche amplification. Here, the specific configuration of the quench resistor 21 may be a MOSFET or another semiconductor element having a quench function. A threshold voltage is set in the buffer 22, and it outputs an H level for a fixed period of time in response to fluctuations in the input voltage, so it has a waveform shaping function. The signal shaped by the buffer 22 in this way is output as a pulse waveform. A counter circuit prepared in the subsequent stage counts the number of pulse waveforms of this pulse signal, and the number of pulses within a fixed period of time is output as a pixel signal value. Here, the circuit configuration of each pixel of the SPAD sensor has been explained based on a simple configuration having a photodiode 20, a quench resistor 21, and a buffer 22, but the pixel circuit configuration may also include other circuit elements or semiconductors.
[0019] (Conversion of output signal to image signal) Next, a flow of processing pixel output signals as image signals will be described with reference to FIG. 3. First, the number of pulses counted for each pixel (corresponding to the number of occurrences of avalanche amplification) is output as an output signal for each pixel. However, since the time resolution of photon counting in a SPAD sensor is limited, when the number of photons in a unit time is extremely large, photons may not be separated one by one and may be counted. As a result, the correlation between the number of pulses and the number of photons has a nonlinear characteristic as shown in FIG. 4(a). Here, the nonlinear characteristic is expressed as a probability function of the number of pulses. There is a nonlinear correlation between the number of photons and the number of pulses, and a linear number of photons can be calculated by converting the observed number of pulses using the inverse function of the nonlinear characteristic. The inverse conversion characteristic that can convert the number of pulses to the number of photons is shown in FIG. 4(b). Therefore, an image signal that is linear with respect to the number of photons can be obtained by the signal processing unit 12 converting the number of pulses (output signal) using the inverse characteristic of the probability function. Next, as one means of exposure control, the brightness of the image is adjusted by amplifying the image signal using digital gain. This makes it possible to obtain an image signal that changes linearly with respect to the number of photons and is maintained at an appropriate brightness. In FIG. 3, linear conversion and digital gain are mentioned as processes to be applied to the number of pulses per pixel, but other processes may be included. For example, FPN correction for detecting and correcting defective pixels, shading correction for correcting unevenness in the black level within the screen, and optical correction for correcting deterioration due to the characteristics of optical lenses may be included. Furthermore, white balance processing for adjusting the WB (white balance) of the image, edge enhancement processing, and noise reduction processing may be included. Moreover, these image processes may be processed by the signal processing unit 12, and some of the processes may be performed within the image sensor 11.
[0020] (Automatic exposure control function) Next, the automatic exposure adjustment function in the control and calculation unit 14 will be described with reference to the flowchart of FIG. 5. First, an image is captured by the image sensor 11 (S500), and the captured image (output signal) is transmitted to the signal processing unit 12, where various image processes are performed. An example of a signal-processed image is shown in FIG. 6(a). In the image, a forest, a backlit building, the sky, and the sun are captured, and the image signal value of the sun is very high, followed by the sky, the forest, and the backlit building, in that order. At this time, the control and calculation unit 14 divides the image into a plurality of regions, acquires a signal value for each region, and evaluates the brightness. FIG. 6(b) shows the signal values of the image signals obtained by dividing the image into four horizontally and four vertically. Here, the signal value obtained from the image region in which the sun is reflected is very large, and the signal value obtained from the image region in which the backlit building is reflected is small. Here, the signal value may be obtained for each color of the Bayer image, or may be obtained for each color and brightness of a YCC or YUV image. Also, the division number of the signal values for each region does not need to be 4 x 4, and it may be 8 x 8 or 16 x 16 regions. Also, while an example of acquiring the average signal value of the image signal for each region has been shown here, it may be acquired as the sum of the signal values within the region. In this way, the signal value is acquired for each region of the image (S501).
[0021] Next, the brightness Y of the subject is evaluated by weighting the signal values acquired for each region (S502). FIGS. 7(a) to 7(c) show examples of weighting tables when evaluating brightness by weighting each region. FIG. 7(a) shows a weighting table when evaluating brightness at the same ratio for all regions without changing the weight for each region. On the other hand, FIG. 7(b) shows a table when evaluating brightness by lowering (making different) the weight of the region at the top of the image, taking into consideration the tendency of a high-luminance subject such as the sun to be reflected in the upper part of the image. FIG. 7(c) shows a weighting table that makes it easier to accurately grasp the brightness of the main subject by increasing the weight of the center of the screen, taking into consideration the tendency of the main subject to be reflected in the center of the image. In other words, the weight of the center of the screen and the weight of the peripheral part of the screen are made different. Here, types of fixed weighting tables for evaluating brightness by changing the weight for each region have been introduced, but the brightness may be evaluated by dynamically switching between these tables depending on the subject and shooting conditions. In addition, other weighting tables may be dynamically generated depending on the type and position of the subject in the image to evaluate the brightness.
[0022] In this way, a signal value indicating the brightness of the image can be calculated from the image signal value of each region obtained from the image and the weighting table. Specifically, the signal value of the image signal of each region is multiplied by the weighting value of each region defined in the weighting table, and the multiplication results of each region are added for all regions, so that the brightness Y of the subject can be evaluated (calculated) (S502). The brightness of the subject can be evaluated by comparing the Y calculated for all regions with Yref, which is a reference value of appropriate brightness previously stored in memory 16. That is, if Y is lower than the reference value Yref, it can be evaluated that the brightness of the subject is insufficient, and if Y is greater than the reference value Yref, it can be evaluated that the brightness is excessive.
[0023] Therefore, ΔY is calculated as the difference between Y and Yref (S503). At this time, it is not necessary for the brightness of the image and the reference value to completely match, and if the difference ΔY is less than a certain threshold value θBv, it is determined that the brightness is within an appropriate range (a predetermined range). Therefore, ΔY=abs(Y-Yref)<θBv (Formula 1) If the above condition is satisfied, Y is determined to be an appropriate brightness (S504). Note that Yref is a reference brightness, and may be, for example, a predetermined ratio to the sum of the signal values of the image signal. Alternatively, it may be a predetermined ratio (for example, 18%) to the maximum signal value of the image signal.
[0024] Next, the automatic exposure adjustment function when the brightness is out of the appropriate range (NO in S504) will be described (S505). When the brightness is out of the appropriate range, the exposure parameters are adjusted in the direction to cancel the brightness by the deviation amount ΔY, so that the brightness can be returned to the appropriate range. In other words, when it is determined that Y, which is the brightness of the image based on the image signal, is out of the specified range, the exposure parameters are controlled. The exposure parameters include aperture, exposure time, digital gain, etc., and which parameters are adjusted is specified by the AE (Auto Exposure) diagram. An example of the AE diagram is shown in FIG. 8. In this AE diagram, when the subject moves from a dark side to a bright side, first the digital gain is minimized to make it darker, then the aperture is used to make it darker. Next, the exposure time is adjusted to make it darker, and a single-density ND filter is inserted halfway through to make the amount of darkness due to the ND filter brighter with the exposure time, and finally the exposure time is used to make it darker. In this way, it is possible to properly control the brightness of the subject by combining multiple exposure parameters by specifying the order of control from the exposure parameters. Here, let us consider an example in which the brightness of the subject is Y1, the result of the brightness evaluation is evaluated as bright by ΔY, and ΔY is determined to be greater than a predetermined threshold value θBv. In FIG. 8, Y1 is in the brightness of the order in which the exposure time is moved, and if the exposure time is darkened by ΔY, it moves to . Here, since the exposure has been moved by ΔY, ΔY=0, and the brightness falls within the appropriate range, so that the automatic exposure adjustment is stopped. It is not necessary to completely set ΔY to 0, and it is sufficient to control the parameters related to the exposure so that ΔY is smaller than the predetermined threshold value θBv. In this way, it is possible to control the brightness of the subject to an appropriate level by evaluating the brightness based on the image signal of each region of the image and mutually controlling the parameters related to the exposure defined by the AE diagram. It is possible to realize the function of the control unit that controls the parameters related to the exposure described above by the control calculation unit 14.
[0025] (Stress change in the image sensor) As explained above, a feature of the SPAD sensor is that a large current flows due to the avalanche amplification phenomenon in response to input photons, and the number of photons can be counted. On the other hand, in order to generate the avalanche amplification phenomenon, it is necessary to apply a reverse bias voltage that exceeds the breakdown voltage, and a large current flows when a large voltage is applied. In the case of using it as an image sensor, in video shooting, 30 frames or more of images are shot per second, and a large current repeatedly flows through the circuit elements of each pixel, which places a very large load. Therefore, the stress of the circuit elements may change due to the repeated large current. If a stress change occurs in the circuit elements of each pixel in the SPAD sensor, there is a risk that the DCR (Dark Count Rate) will increase. Therefore, if the number of avalanche amplifications of each pixel of the SPAD sensor is accumulated and counted, the DCR generated in each pixel can be roughly predicted. However, when the accumulated value of the number of avalanche amplifications is stored in the sensor on a pixel-by-pixel basis, the amount of data to be stored increases as the number of pixels on the sensor increases, making it difficult to continue storing the value in the counter circuit of the sensor.
[0026] (Calculation of output signal and storage of accumulated value) Therefore, a case where the signal values of the output signal are acquired and accumulated by region from the captured image is considered. The output signal indicates the number of occurrences of avalanche amplification (i.e., the number of pulses). In this embodiment, the output signal and the image signal are distinguished as different signals. The output signal is a signal output from the image sensor 11, and the image signal is a signal converted and processed by the signal processing unit 12. The signal values of the output signal may be acquired for each color of the Bayer image, or may be acquired for each color and brightness of the YCC or YUV image. The division number of the signal values by region does not need to be 4×4, and may be acquired in 8×8 or 16×16 regions. In addition, an example of acquiring the signal values of the image signal by region using the automatic exposure adjustment function has been shown, but the signal values of the image signal by region may be used in the same way as the automatic exposure adjustment. In addition, an example of acquiring the average value of the signal values of the output signal by region has been shown here, but the signal values may be acquired as the sum of the signal values of the output signal in the region.
[0027] As mentioned above, the image signals obtained for each region are subjected to several conversion processes from the output signal (number of pulses). Therefore, the signal value of the image signal is different from the number of avalanche amplifications. In the flow shown in Figure 3, digital gain and linear conversion are performed, but in order to calculate the output signal from the image signal, the applied conversion must be restored. Below, we will explain the flow of the process of calculating the cumulative value of the number of avalanche amplifications (i.e., the signal value of the output signal) from the image signal with reference to the flowchart in Figure 9.
[0028] First, an image is captured using a SPAD sensor (S900). A linear transformation and a digital gain are applied to the captured image (output signal), and an image signal for each region is acquired (S901). Next, the acquired image signal is inversely transformed to the amount amplified by the digital gain (S902). Furthermore, the inversely transformed image signal is inversely transformed based on the nonlinear characteristic of the pulse number-image signal shown in FIG. 4(b) (S903). Note that the output signal from the SPAD sensor may not be equivalent to the number of occurrences of avalanche amplification. For example, it is quite conceivable that a signal offset relative to the pulse number in the sensor is output as the output signal. In that case, the signal processing unit 12 can more accurately determine the number of occurrences of avalanche amplification by subtracting the signal value for the amount offset in the sensor in addition to the above two inverse transformation processes. In this way, the number of occurrences of avalanche amplification can be calculated by performing a predetermined transformation including at least one of the inverse transformation and the subtraction process on the image signal. However, the offset in the sensor may be allowed, and the output signal may be regarded as the number of occurrences of avalanche amplification as it is. In this case, the steps of S902 and S903 can be omitted by storing the output signal before the conversion process by the signal processing unit 12 as an accumulated value. The function of the calculation unit that calculates the number of occurrences of avalanche amplification in the image sensor 11 based on the image signal is realized by the signal processing unit 12 or the control and calculation unit 14. The number of occurrences of avalanche amplification (output signal, number of pulses) obtained by inverse conversion (calculation) is accumulated by region and stored in the memory 16 as a storage unit while the camera is activated and images are being captured (S904). That is, the accumulated value of the number of occurrences of avalanche amplification calculated by the calculation unit (signal processing unit 12 or control and calculation unit 14) is stored in the storage unit (memory 16). The memory 16 or an auxiliary storage device (not shown) functions as a storage unit that stores the accumulated value of the number of occurrences of avalanche amplification calculated by the calculation unit.
[0029] By calculating and storing the number of times avalanche amplification occurs for each captured image by the calculation unit, the number of times avalanche amplification occurs can be grasped for each region, and the effect of stress changes in the circuit elements (increase in DCR) can be estimated.
[0030] Here, since the cumulative value can be an astronomical figure depending on the brightness of the subject of the camera and the operation period, the cumulative number may be stored in a logarithmic representation or a floating-point representation as a method for holding the cumulative number. Also, it may be stored by dividing it into an exponent part and an integer part. Also, the value stored as the cumulative value may not be the number of pulses itself, but may be other physical quantities from which the number of pulses can be calculated. For example, the cumulative value of the image signal for each region may be stored in the memory 16, and the cumulative value of the image signal may be converted into the cumulative value of the output signal at the timing when the stored cumulative value is referred to. Alternatively, it may be stored as a difference value or a ratio value from a reference value (for example, Yref, etc.), or a value obtained by logarithmically converting the ratio value. In this case, it is possible to convert into the number of pulses using these physical quantities and the reference value, and to estimate the degree of stress change based on the number of pulses.
[0031] Also, the Ev value, which represents the absolute brightness obtained by adding the Av, Tv, and Sv values to the image signal value, may be stored for each region. In this case, the ΔBv value based on the reference value is calculated by subtracting the accumulated Av, Tv, and Sv values from the accumulated Ev value, so the number of pulses can be estimated using the reference value.
[0032] Furthermore, regarding the accumulation of pulse counts, in applications such as surveillance cameras that continuously capture images of a fixed subject for a long period of time, it is not necessary to accumulate the number of pulses for all consecutive frames; it is acceptable to calculate and accumulate the number of pulses for each area in the image at regular intervals.
[0033] (Exposure control based on cumulative values) The correlation between the number of avalanche amplifications and DCR is shown in Figure 10. In Figure 10, the horizontal axis shows the number of avalanche amplifications, but here we assume that the number is very large, on the order of mega or giga. The vertical axis shows the amount of noise due to dark current per unit area that occurs in the image due to deterioration. When the amount of noise increases, the brightness evaluation due to exposure change may differ. Figure 11 shows two examples where the same brightness evaluation results are obtained when there is a lot of noise ((a)-1) and when there is little noise ((a)-2). Here, if we consider brightening by adjusting the aperture or exposure time, the brightness of the noise does not change by controlling the aperture or exposure time, so only the actual brightness of the subject minus the amount of noise increases. Therefore, the two examples that were the same brightness before brightening do not become the same brightness after changing the exposure, as shown in Figure 11 (b)-1 and (b)-2.
[0034] Next, consider an example of changing exposure with digital gain. Figure 11 shows two examples where the same brightness evaluation result is obtained when there is a lot of noise ((a)-1) and when there is little noise ((a)-2). When making the image brighter with digital gain, the image signal containing noise is digitally amplified, so the noise is also amplified. As a result, the image quality deteriorates as the noise is amplified, but the same brightness evaluation result is obtained. In this way, the effect of deterioration due to the number of avalanche amplifications varies depending on the exposure-related parameters controlled during automatic exposure adjustment. In other words, if the exposure is controlled to move significantly in accordance with the exposure changes of the aperture and exposure time, the exposure will move too much and become unstable when the exposure is controlled with digital gain. Conversely, if the amount of exposure control is reduced in accordance with the exposure changes with digital gain, it will take a long time to reach the appropriate exposure in the aperture and gain range, or it will stop short of reaching the appropriate exposure.
[0035] Therefore, we consider using the cumulative value of the number of occurrences of avalanche amplification (hereinafter, the cumulative value). Figure 13 shows a flowchart when the cumulative value is used for exposure control. It is known that the cumulative value is correlated with the change in dark current in each area of the SPAD sensor. Therefore, the cumulative value is obtained by area by referring to the storage unit (memory 16) (S1300). Figure 14 shows an example of the cumulative value by area. Note that M represents 10^6. If the threshold T at which the change in dark current begins to become noticeable is 150M, then the increase in dark current is noticeable in 8 areas out of 16 areas that exceed 150M, which is 50% of the area. In this way, as one example, the percentage of areas that are equal to or greater than the threshold T at which the change becomes noticeable is calculated (S1301).
[0036] We have explained that there are cases where changing the exposure does not result in a change in brightness compared to the changed exposure amount due to the influence of noise. Here, we will explain a method for achieving stable exposure control by utilizing the proportion of the area where the cumulative value is equal to or greater than the threshold value T.
[0037] First, in the control of aperture and exposure time, the change in the amount of light is reduced compared to the control of digital gain, so the automatic exposure adjustment takes time. Therefore, we consider increasing the exposure response coefficient to restore the time required for automatic exposure adjustment. The exposure response coefficient is a coefficient that specifies how much unit time it takes to reach the target exposure difference. Based on this coefficient, the amount of exposure change during one frame is determined. Figure 15 is a relationship diagram showing the relationship between the proportion of the area where the cumulative value is equal to or greater than the threshold T and the exposure response coefficient. When the proportion of the area where the cumulative value is equal to or greater than the threshold T is small, it is 0.5 [EV / sec], which indicates a response of 0.5 seconds to change the exposure by one step. On the other hand, when the proportion of the area where the cumulative value is equal to or greater than the threshold T becomes large, the change in the amount of light for the changed exposure decreases, so the change in the amount of light can be increased by increasing the exposure response coefficient. Now, as shown in Figure 14, if the proportion of the area where deterioration is noticeable is 50%, the exposure response is found to be 0.65 from the relationship diagram in Figure 15. In this way, the exposure response coefficient is determined based on the cumulative number of occurrences of avalanche amplification (S1302), which makes it possible to prevent the exposure from taking a long time to stop.
[0038] On the other hand, if the amount of exposure that moves at one time is increased, the amount of exposure that moves in one frame increases, so the range of the appropriate exposure may be exceeded, and the exposure may not converge. Although the increase in dark current is shown in FIG. 10, the change does not necessarily follow this pattern. In other words, the stress changes with respect to the amount of input light with the number of occurrences of avalanche amplification, but the degree of change in stress may vary. As a result, there are various possibilities for the change in the signal value on the image with respect to the amount of exposure change, and the exposure response may return to normal or may move too fast. Therefore, the threshold value θBv used to determine the appropriate exposure is changed in a wider direction. In other words, the range in which the brightness is determined to be appropriate (the range in which the exposure-related parameters are not changed) is determined based on the cumulative value of the number of occurrences of avalanche amplification. Specifically, the value of the above-mentioned threshold value θBv is increased (the range in which the brightness is determined to be appropriate is expanded). Furthermore, as can be derived from FIG. 16, the value of θBv is increased as the proportion (or number) of the area in which the cumulative value is equal to or greater than the threshold value T increases (the range in which the brightness is determined to be appropriate is expanded). As a result, even if the amount of exposure movement in one frame increases, it will remain within the appropriate range, and automatic exposure adjustment can be stopped stably. Figure 16 shows the relationship between the ratio of the area where the cumulative value is equal to or greater than the threshold value T and the threshold value (range) for appropriateness judgment. When the ratio was low, θBv = 0.3 [steps], but to prevent instability as a result of increasing the response, it can be updated to θBv = 0.45 [steps] (S1303), allowing shooting with stable exposure. In this way, by changing the exposure response coefficient and the threshold value (range) for appropriateness judgment according to the degree of deterioration, stable exposure control can be achieved even if the characteristics of the SPAD sensor change.
[0039] In the above description, the exposure control using the ratio of the area where the number of avalanche amplifications is equal to or greater than the threshold value has been described, but other indices may be used. For example, the maximum value of the accumulated value may be used, or the accumulated value of the number of avalanche amplifications occurring in a specific area such as the center of the image may be used. In other words, the coefficient of the exposure response and the threshold (range) of the suitability judgment are determined based on the maximum value of the accumulated value of the number of avalanche amplifications occurring. Alternatively, the coefficient of the exposure response and the threshold (range) of the suitability judgment are determined based on the accumulated value of the number of avalanche amplifications occurring in a specific area (for example, the center or a specified area). In addition, the accumulated value of the number of avalanche amplifications occurring may be stored for each pixel, and the coefficient of the exposure response and the threshold (range) of the suitability judgment may be determined based on the area ratio or the number of pixels where the accumulated value is equal to or greater than the threshold value T. In addition, the accumulated value may be reset due to reasons such as replacing the sensor, in which case it is possible to reset the coefficient of the exposure response to the original value or reduce the threshold (range) of the suitability judgment.
[0040] In this embodiment, the case where the exposure response coefficient and the threshold (range) for suitability judgment are changed based on the cumulative value of the number of occurrences of avalanche amplification have been described, but either one of them may be changed. For example, in the flowchart of Fig. 13, S1302 may not be performed, and only S1303 may be performed. Even with only S1303, the threshold θBv is changed so that the brightness Y does not erroneously exceed the threshold for suitability judgment due to an increase in dark current, and therefore the effect of stabilizing AE can be obtained.
[0041] The same applies to the relationship between the coefficient of the exposure response and the proportion of the area where the cumulative value is equal to or greater than the threshold T. Although a linear relationship is illustrated in FIG. 15, the relationship may be a nonlinear relationship or a multidimensional function. The relationship between the proportion of the area where the cumulative value is equal to or greater than the threshold T and the threshold (range) of the appropriateness determination shown in FIG. 16 is also similar, and other relationships may be used. This is merely a diagram showing one example of the relationship with the proportion of the area where the cumulative value is equal to or greater than the threshold T. Therefore, this does not limit either the relationship between the proportion of the area where the cumulative value is equal to or greater than the threshold T and the exposure response, or the proportion of the area where the cumulative value is equal to or greater than the threshold T and the threshold (range) of the appropriateness determination.
[0042] <Other embodiments> Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.
[0043] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions. [Explanation of symbols]
[0044] 10. Imaging optical system 11 Image sensor 12 Signal Processing Section 13 Video output section 14 Control and calculation section 15 CPU 16 Memory
Claims
1. an imaging element configured with an avalanche photodiode for photoelectrically converting an optical image; an image processing unit that generates an image signal based on an output signal from the imaging element; a calculation unit that calculates the number of occurrences of avalanche amplification in the imaging element based on the image signal; a storage unit that stores the cumulative value of the occurrence count calculated by the calculation unit; a control unit that controls a parameter related to the exposure when it is determined that the brightness of an image based on the image signal is outside a predetermined range, The imaging apparatus, wherein the control unit determines the predetermined range based on the accumulated value stored in the storage unit.
2. 2. The imaging device according to claim 1, wherein the number of occurrences is calculated by performing a predetermined conversion on the image signal.
3. 3. The imaging device according to claim 2, wherein the predetermined transformation includes at least one of an inverse transformation process of a digital gain, an inverse transformation process for an inverse transformation of a linear transformation of a characteristic of the imaging element, and a subtraction process for subtracting a signal value offset by the imaging element.
4. The imaging device according to claim 1 , wherein the occurrence count is calculated for each of a plurality of regions.
5. The imaging device according to claim 1 , wherein the cumulative value of the number of occurrences is stored for each of the plurality of regions.
6. The imaging device according to claim 1 , wherein the control unit calculates the number or the ratio of areas where the cumulative value of the occurrence count is equal to or greater than a predetermined threshold value.
7. The imaging device according to claim 6 , wherein the control unit determines the predetermined range based on the number or the ratio.
8. 7. The imaging device according to claim 6, wherein the predetermined range is determined such that the greater the number or the ratio, the wider the predetermined range becomes.
9. 2. The imaging device according to claim 1, wherein the brightness of the image is determined to be outside the predetermined range based on a difference between a signal value based on the image signal, which indicates the brightness of the image, and a reference value.
10. The image pickup apparatus according to claim 1 , wherein the control unit determines an exposure response coefficient based on the accumulated value stored in the storage unit.
11. 2. The exposure control device according to claim 1, wherein the predetermined range is determined based on a maximum value of the number of occurrences for each of the plurality of regions.
12. 6. The imaging device according to claim 5, wherein the predetermined range is determined based on the cumulative value of the occurrence count for a specific area among the cumulative values of the occurrence counts stored for the plurality of areas.
13. 2. The exposure control device according to claim 1, wherein the exposure-related parameters include at least one of an aperture, an exposure time, and an ND filter.
14. an imaging element configured with an avalanche photodiode for photoelectrically converting an optical image; a calculation unit that calculates the number of occurrences of avalanche amplification in the image sensor based on an output signal from the image sensor; a storage unit that stores the cumulative value of the occurrence count calculated by the calculation unit; a control unit that controls a parameter related to the exposure when it is determined that the brightness of the image captured by the image capture element is outside a predetermined range, The imaging apparatus, wherein the control unit determines the predetermined range based on the accumulated value stored in the storage unit.
15. an image processing step of generating an image signal based on an output signal from the imaging element; a calculation step of calculating the number of occurrences of avalanche amplification in the imaging element based on the image signal; a storage step of storing the cumulative value of the occurrence count calculated in the calculation step; a control step of controlling the exposure-related parameters when it is determined that the brightness of the image based on the image signal is outside a predetermined range, The control method is characterized in that, in the control step, the predetermined range is determined based on the cumulative value stored in the storage step.
16. A program for causing a computer to execute the control method according to claim 15.
17. A computer-readable storage medium storing the program according to claim 16.