Imaging apparatus, imaging method, and computer program

Abstract

To provide an imaging apparatus capable of stably adjusting exposure even when characteristics of an avalanche photodiode are changed.SOLUTION: An imaging apparatus includes an image sensor including an avalanche photodiode, an image processor configured to generate an image signal based on an output of the image sensor, a calculator configured to calculate the number of times of occurrence of avalanche amplification in the image sensor, a storage configured to store a cumulative value of the number of times of occurrence for each region of the image signal, and an evaluator configured to evaluate brightness of an image based on the cumulative value.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to an imaging device, an imaging method, a computer program, and the like. [Background technology]

[0002] One type of image sensor is the Single Photon Avalanche Diode (SPAD) sensor (hereafter referred to as the SPAD sensor). SPAD sensors utilize the avalanche amplification phenomenon, in which electrons are accelerated by applying a strong electric field, and then collide with other electrons, creating multiple electrons, causing an avalanche-like phenomenon that generates a large current.

[0003] This allows weak photons incident on the pixel to be converted into a large current and detected as an electric charge. Because of its mechanism, SPAD sensors are expected to be used as image sensors because they do not introduce noise during signal readout. In particular, because they can capture subjects clearly without being affected by noise even in dark places, they are expected to be widely used as image sensors for surveillance and other applications.

[0004] In addition, when considering surveillance applications, it is desirable that the exposure be automatically adjusted according to the brightness of the subject, since the camera will be set up in a fixed location and continue to capture images for a long period of time. [Prior art documents] [Non-patent literature]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-25171 Summary of the Invention [Problem to be solved by the invention]

[0006] Patent Document 1 discloses a photoelectric conversion device having an APD (Avalanche Photo Diode). It also discloses that when the APD is applied to an image sensor equipped with a general Bayer array color filter, the number of photons incident on the APD varies for each pixel due to differences in light transmittance, resulting in changes in sensitivity and saturation.

[0007] It is also known that the dark current of an APD can change or increase depending on the number of times the avalanche amplification phenomenon occurs. Therefore, if the number of photons incident on the APD varies from pixel to pixel, the degree to which the dark current changes or increases may differ from pixel to pixel. In this case, the brightness of the subject may not be captured correctly depending on the level of the dark current.

[0008] Therefore, when controlling exposure by giving priority to an important area or a specific color within the angle of view, it may not be possible to accurately measure the brightness of the important area or the specific color. Also, automatic exposure control may cause the optimum brightness to be exceeded multiple times, making it impossible to stop automatic exposure and resulting in instability.

[0009] The present invention has been made in view of the above problems, and one of its objects is to provide an imaging device that allows stable exposure adjustment even if the characteristics of the avalanche photodiode change. [Means for solving the problem]

[0010] An imaging device according to one aspect of the present invention comprises: an imaging element having an avalanche photodiode; an image processing unit that generates an image signal based on an output of the imaging element; a calculation unit that calculates the number of occurrences of avalanche amplification in the imaging element; a storage unit that stores the cumulative value of the number of occurrences for each region of the image signal; and an evaluation unit that evaluates the brightness of the image based on the cumulative value. [Effects of the Invention]

[0011] According to the present invention, an imaging device that allows stable exposure adjustment even if the characteristics of the avalanche photodiode change can be realized. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an example of the internal configuration of an imaging device 100 according to a first embodiment. [Figure 2] 1 is an equivalent circuit diagram of a pixel that constitutes an imaging element according to Embodiment 1. FIG. [Figure 3] 4 is a flowchart showing an example of image signal processing according to the first embodiment. [Figure 4] 10A and 10B are graphs showing an example of the relationship between the number of input photons, the number of pulse counts, and the image signal value according to the first embodiment. [Figure 5] 1 is a flowchart showing an example of brightness evaluation processing according to the first embodiment. [Figure 6] 1A and 1B are diagrams showing examples of captured images and signal values ​​of the captured images according to the first embodiment. [Figure 7] 10(A) to 10(C) are diagrams showing examples of weighting tables used for evaluating brightness according to the first embodiment. [Figure 8] 10A and 10B are diagrams showing an example of a UI for presenting a brightness evaluation result to a user according to the first embodiment. [Figure 9] 5 is a flowchart showing an example of a process for calculating a cumulative count value from an image signal in the first embodiment. [Figure 10] 10 is a graph showing an example of the correlation between the number of avalanche amplifications and the amount of dark current noise generated per unit area according to the first embodiment. [Figure 11] 5A to 5C are diagrams showing examples of changes in image brightness when exposure is controlled by aperture or exposure time in the first embodiment. [Figure 12] 5A and 5B are diagrams illustrating an example of changes in image brightness when exposure is controlled using digital gain in the first embodiment. [Figure 13]FIG. 4 is a diagram showing an example of cumulative count values ​​of avalanche amplification for each region according to the first embodiment. [Figure 14] FIG. 10 is a diagram showing an example of the relationship between the cumulative count value for each region and the subtraction rate of the weighting coefficient when evaluating brightness in step S502. [Figure 15] 10(A) to 10(C) are diagrams showing examples of corrected weight tables according to the first embodiment. [Figure 16] 10 is a flowchart showing an example of a process for evaluating brightness using a color signal according to the second embodiment. [Figure 17] FIG. 10 is a diagram showing an example of the relationship between the correction coefficients used in determining color signal selection and the cumulative count values ​​for each region according to the second embodiment. [Figure 18] 10(A) to 10(C) are diagrams showing examples of color signal values, cumulative values, and corrected color signal values ​​in a certain area according to the second embodiment. [Figure 19] 11A and 11B are diagrams showing an example of setting a region of interest in the third embodiment. [Figure 20] FIG. 11 is a diagram showing an example of the relationship between the cumulative count value by region and the weighting coefficient α according to the third embodiment. [Figure 21] 10A to 10C are diagrams showing an example of resetting a region of interest according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions will be omitted or simplified.

[0014] <Embodiment 1> Fig. 1 is a block diagram showing an example of the internal configuration of an image capturing apparatus 100 according to embodiment 1. Note that some of the functional blocks shown in Fig. 1 are realized by causing a CPU or the like serving as a computer included in the image capturing apparatus 100 to execute a computer program stored in a memory serving as a storage medium.

[0015] However, some or all of these functions may be implemented by hardware. Examples of hardware that can be used include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, the functional blocks shown in Figure 1 do not have to be built into the same housing, and may be configured as separate devices connected to each other via signal paths.

[0016] The imaging optical system 10 includes a zoom lens, a focus lens, an anti-vibration lens, an aperture, an ND (Neutral Density) filter, etc. Each component of the imaging optical system 10 is controlled to determine the angle of view, focus, zoom, exposure amount, etc. when capturing an image of a subject. The imaging optical system 10 forms an optical image of the subject on the imaging surface of an imaging element 11.

[0017] The image sensor 11 has a plurality of pixels, each of which performs photoelectric conversion to convert an optical image formed on the imaging surface into an electrical signal. Each pixel constituting the image sensor 11 of this embodiment has an avalanche photodiode for photoelectrically converting the optical image, and functions as a SPAD sensor that counts the number of incident photons. The output signal from the image sensor 11 is output to a signal processing unit 12, where various image processing operations are performed.

[0018] The signal processing unit 12 functions as an image processing unit and executes image processing steps for generating an image (image signal) based on the output signal from the imaging element 11. The signal processing unit executes various correction processes such as removal of fixed pattern noise contained in the output signal from the imaging element 11, brightness correction using digital gain, demosaicing processing, WB (white balance) correction processing, edge enhancement processing, and gamma processing.

[0019] Furthermore, the signal processing unit 12 performs recognition processing to detect the subject area from the image (image signal) and generates control signals for controlling each unit of the imaging optical system 10. The signal processing unit 12 also generates evaluation values ​​for exposure control and WB correction, and the generated evaluation values ​​are supplied to the control calculation unit 14. The image corrected by the signal processing unit 12 is transmitted to the video output unit 13.

[0020] The video output unit 13 outputs the corrected image together with a synchronization signal to an external device via an output terminal (not shown) or a network.

[0021] The output terminal may be any terminal capable of transmitting and receiving video signals to and from an external device. Terminals such as SDI (Serial Digital Interface), HDMI (registered trademark) (High-Definition Multimedia Interface), USB (Universal Serial Bus), and RJ45 are acceptable. Terminals conforming to a unique standard for transmitting and receiving signals to and from specific external devices are also acceptable.

[0022] The control and calculation unit 14 generates control information to be supplied to the imaging optical system 10, the imaging element 11, the signal processing unit 12, and the video output unit 13. The control and calculation unit 14 includes a CPU 15 as a computer, and the CPU 15 controls each unit of the imaging device 100 by executing a computer program stored in the memory 16 or an auxiliary storage device (not shown).

[0023] The memory 16 stores data and computer programs necessary for calculating the control information, and stores data during calculations and calculation results in the memory 16. The CPU 15 appropriately refers to the memory 16 while calculations are performed in the control calculation unit 14.

[0024] The image signal generated by the signal processing unit 12 is stored in a memory 16 via a control and calculation unit 14. Although the memory 16 is herein considered to be a part of the imaging device 100, it may be configured to be detachable from the imaging device 100.

[0025] The operation display unit 17 receives operations from the user and displays various information. It can also receive information from the user regarding the settings of the imaging device 100. The operation display unit 17 can also receive and display setting information for the imaging device 100 from the control and calculation unit, and can also receive and display images processed by the signal processing unit 12 via the control and calculation unit 14.

[0026] In addition, the operation display unit 17 of this embodiment will be described using an example of a display (touch panel) with a touch operation detection mechanism that can simultaneously accept operations and display information, but the operation member and the display member may be separate members.

[0027] 2 is an equivalent circuit diagram of a pixel constituting the image sensor according to embodiment 1, and describes an example of a pixel having a SPAD sensor. Each pixel is composed of a photodiode 20, a quench resistor 21 capable of controlling a quench voltage VQ, and a buffer 22, and the photodiode 20 is a SPAD.

[0028] When a reverse voltage exceeding the breakdown voltage is applied to the photodiode 20, it operates in Geiger mode, triggered by the input of a photon. Therefore, a large reverse bias voltage equal to or greater than the breakdown voltage of the voltage VH applied to the SPAD is applied.

[0029] When a photon enters the photodiode 20, an avalanche current is generated. The quench resistor 21 is configured so that its resistance value changes according to the quench voltage VQ applied to the control terminal. The quench resistor 21 reduces the reverse bias voltage of the avalanche current generated in the photodiode 20, making it possible to control the end of avalanche amplification.

[0030] It should be noted that a MOSFET or other semiconductor element having a quench function may be used as the quench resistor 21. A threshold voltage is set in the buffer 22, and the buffer 22 shapes the waveform by outputting an H level for a certain period of time in response to fluctuations in the input voltage.

[0031] This causes a pulse signal to be output from the buffer 22. Furthermore, the number of pulses is counted by a counter circuit (not shown) connected to the buffer 22, and the number of pulses within a certain period is output as a pixel signal value according to the amount of incident light.

[0032] Here, we have explained an example of a simple circuit configuration for each pixel of the SPAD sensor, which has a photodiode 20, a quench resistor 21, and a buffer 22, but the pixel circuit configuration may also include other circuit elements or semiconductors.

[0033] 3 is a flow diagram showing an example of image signal processing according to embodiment 1. First, the number of pulses counted for each pixel (corresponding to the number of occurrences of avalanche amplification) is output as an output signal for each pixel. In the example of image signal processing in FIG. 3, the number of pulses is linearly converted, and then digital gain processing is performed.

[0034] However, since SPAD sensors have a limit to the time resolution of photon counting, if there are an extremely large number of photons within a unit time, they may not be able to separate each photon individually and may end up counting them. 4A and 4B are graphs showing an example of the relationship between the number of input photons, the number of pulse counts, and the image signal value according to embodiment 1. In the SPAD sensor described above, the correlation between the number of input photons and the number of pulse counts has nonlinear characteristics as shown in FIG.

[0035] This nonlinear characteristic is expressed as a probability function of the number of pulses. Thus, there is a nonlinear correlation between the number of input photons and the number of pulse counts. Therefore, for example, in the digital gain processing in Figure 3, the observed number of pulse counts is converted by the inverse function of the nonlinear characteristic in Figure 4(A). This makes it possible to calculate an image signal value that is linear with respect to the number of photons.

[0036] Figure 4(B) shows an example of the inverse conversion characteristic for converting the pulse count number into an image signal value in the digital gain processing of Figure 3. In the digital gain processing of Figure 3, by converting the pulse count (output signal) into a pixel signal value using the inverse conversion characteristic, which is the inverse characteristic of the probability function, as shown in Figure 4(B), it is possible to obtain an image signal that is linear with respect to the number of photons.

[0037] In this embodiment, brightness is evaluated by the image capture device 100, and digital gain processing is performed for exposure control based on the evaluation result. In this case, the digital gain processing in Fig. 3 may be performed using the inverse conversion characteristics described above. This allows the image signal value to be made linear with respect to the number of photons while adjusting the exposure.

[0038] In addition, the image signal processing shown in FIG. 3 may further include, for example, FPN correction for detecting and correcting defective pixels, shading correction for correcting unevenness in the black level within the screen, and optical correction for correcting degradation due to the characteristics of the optical lens.

[0039] In addition, white balance processing for adjusting the WB (white balance) of the image, edge enhancement processing, and noise reduction processing may be performed. These image processing operations may be performed by the signal processing unit 12, or a part of the processing may be performed inside the image sensor 11.

[0040] Fig. 5 is a flowchart showing an example of brightness evaluation processing according to embodiment 1. Note that the operation of each step in the flowchart of Fig. 5 is performed sequentially by a CPU or the like serving as a computer in the control and calculation unit 14 executing a computer program stored in memory.

[0041] 5 starts when the imaging device 100 is activated. Next, in step S500, an image is captured by the imaging element 11. The captured image is transmitted to the signal processing unit 12, where various image processes are performed.

[0042] 6(A) and (B) are diagrams showing examples of captured images and signal values ​​of the captured images according to the first embodiment, with an example of a signal-processed image being shown in FIG. 6(A). The image shown in FIG. 6(A) includes a mountain, a backlit building, the sky, and the sun, with the sun having a very high image signal value, followed by the sky, the mountain, and the backlit building, with the image signal values ​​decreasing in that order. In step S501, the control and calculation unit 14 divides the image into multiple regions and acquires the signal value of each region.

[0043] FIG. 6B shows an example of signal values ​​of image signals obtained by dividing the image of FIG. 6A into four regions horizontally and four regions vertically.

[0044] In this example, the signal value obtained from the image area where the sun is reflected is very large, while the signal value obtained from the image area corresponding to the backlit building is small. Here, the signal value refers to the signal from each pixel of the image sensor, or, for example, the luminance signal in a YCC image signal or YUV image signal.

[0045] Furthermore, the number of divisions into signal values ​​for each region does not need to be 4 x 4, and may be 8 x 8, 16 x 16, or even more. In this way, in step S501, signal values ​​are obtained for each region of the image, and in step S502, the signal values ​​of each region are weighted and added to calculate an added value Y, and the brightness of the subject is evaluated based on the added value Y. That is, in step S502, the image signal is weighted for each region to evaluate the brightness.

[0046] In the brightness evaluation in step S502, the brightness of the image is evaluated based on the cumulative value of the number of occurrences of avalanche amplification in the image sensor, as will be described later. That is, step S502 functions as an evaluation step (evaluation unit) that evaluates the brightness of the image based on the cumulative value.

[0047] 7(A) to 7(C) are diagrams showing examples of weighting tables used in evaluating brightness according to embodiment 1, and show examples of weighting tables when evaluating brightness by weighting each region. Fig. 7(A) shows a weighting table when evaluating brightness for all regions at the same ratio.

[0048] On the other hand, Fig. 7(B) shows an example of a table for evaluating brightness by lowering the weight of the upper region of the image, taking into account the tendency for high-brightness objects such as the sun to appear at the top of the image. Fig. 7(C) shows an example of a weighting table for making it easier to optimize the brightness of the main subject by increasing the weight of the center of the screen, taking into account the tendency for the main subject to appear at the center of the image.

[0049] 7(A) to 7(C) show several types of fixed weighting tables for evaluating brightness by changing the weight for each region, but these tables may be dynamically switched depending on the subject and shooting conditions to evaluate brightness. Also, other weighting tables may be dynamically generated depending on the type and position of the subject in the image to evaluate brightness.

[0050] In this way, the brightness of the subject can be weighted and evaluated based on the signal value of the image signal for each region acquired from the image and the weighting table. Specifically, the signal value of the image signal for each region is multiplied by the weighting value for each region defined in the weighting table, and the multiplication results for each region are added together across all regions, thereby enabling a weighted evaluation of the brightness of the subject.

[0051] In addition, the sum Y, which is the result of adding up the entire area, is compared with Yref, which is a reference value of appropriate brightness stored in advance in memory 16, and if Y is lower than the reference Yref, it is determined that the brightness is insufficient.

[0052] Conversely, if Y is greater than the reference value Yref, it is determined that the brightness is excessive. Therefore, in step S503, ΔY, which is the difference between Y and Yref, is calculated using the following equation 1. ΔY=Y-Yref (Equation 1)

[0053] Note that Yref is a reference brightness, and may be, for example, a predetermined percentage of the total signal value of the image signal. Alternatively, it may be a predetermined percentage (e.g., 18%) of the maximum signal value of the image signal. ΔY calculated in this way can be presented to the user as the deviation from the appropriate exposure state.

[0054] 8A and 8B are diagrams showing an example of a UI for presenting the brightness evaluation result to the user according to the first embodiment, and show an example of presenting the result to the user using the UI of the operation display unit 17.

[0055] Figure 8(A) shows an example in which the result of brightness evaluation is shown to the user by flashing an indicator showing the current brightness, with the center of the linear scale being appropriate brightness, from the center to the right being too bright, and from the center to the left being not bright enough. Figure 8(B) shows an example in which ΔY is presented to the user as a numerical value.

[0056] Next, in step S504, based on the brightness evaluation result, automatic exposure adjustment is performed to achieve appropriate brightness, and then the process returns to step S500 to repeat the flow in Fig. 5. In step S504, if the difference |ΔY| is within a predetermined appropriate range, it is determined that the brightness is appropriate.

[0057] On the other hand, if the image falls outside the optimum range, the exposure parameters are adjusted to bring it back into the optimum range. That is, at least one of the following is controlled: the aperture, the exposure time (charge accumulation time) of the image sensor, the digital gain of the image processing unit, or the insertion or removal of an ND (Neutral Density) filter. The aperture and ND filter are used to control the amount of light incident on the image sensor.

[0058] The order in which the parameters are adjusted is determined by a predetermined AE (Auto Exposure) diagram. Here, step S504 functions as an exposure control step (exposure control section) that controls at least one of the amount of light incident on the image sensor, the exposure time of the image sensor, and the gain of the image processing section based on the brightness evaluation.

[0059] A feature of SPAD sensors is that they can count the number of photons by generating a large current due to the avalanche amplification phenomenon in response to input photons, but to generate the avalanche amplification phenomenon, a reverse bias voltage exceeding the breakdown voltage must be applied. In other words, a large voltage must be applied, which causes a large current to flow.

[0060] When considering applications as an image sensor, video capture requires capturing 30 or more frames per second, causing a large current to repeatedly flow through each pixel, placing a heavy load on the device, and the repeated large currents can cause changes in the stress on the circuit elements.

[0061] If stress changes occur in the circuit elements of each pixel in a SPAD sensor, the signal value (number of pulses) of the output signal for the same amount of light will become uneven from pixel to pixel, and even the pixels will be destroyed, resulting in fixed pattern noise, which will have a negative impact on the image.

[0062] Furthermore, this stress change is not simply proportional to the change over time, but progresses according to the number of times a large current flows due to avalanche amplification. In other words, the degree of pixel degradation varies depending on the brightness of the subject being continuously photographed.

[0063] Therefore, in this embodiment, the number of avalanche amplification events for each pixel of a SPAD sensor is accumulated and counted, thereby making it possible to grasp the degree of pixel degradation. However, when a SPAD sensor is used as an image sensor, the number of pixels on the sensor is large, and accumulating the number of avalanche amplification events on a pixel-by-pixel basis results in a large amount of data, making it difficult to continuously store this data.

[0064] Therefore, in this embodiment, the signal values ​​of the output signals are acquired for each region of the captured image and accumulated, and an output signal indicating the number of occurrences of avalanche amplification (i.e., the number of pulses) for each region is acquired. Also, in this embodiment, the output signals are processed separately from the image signals.

[0065] The output signal is a signal output from the image sensor 11, and the image signal is a signal converted by the signal processing unit 12. When a color filter is arranged in each pixel of the image sensor in a Bayer arrangement, for example, the signal value of the output signal may be acquired for each color of the image, or may be acquired for each color and brightness of a YCC image signal or a YUV image signal.

[0066] Furthermore, the number of divisions into which the signal values ​​are divided for each region does not have to be 4 x 4, and may be 8 x 8, 16 x 16, or even more. Furthermore, as described with reference to Figures 5 to 7, when performing automatic exposure adjustment by acquiring the signal value of the image signal for each divided region, the signal value of the same divided region as the divided region used in the automatic exposure adjustment may be used.

[0067] However, the signal value of the image signal will be different from the number of avalanche amplifications because the image signal obtained for each divided region has been subjected to image processing as explained in Fig. 3. For example, in the flow shown in Fig. 3, linear conversion and the conversion processing shown in Fig. 4(B) in digital gain processing have been performed, so in order to calculate the output signal (number of avalanche amplifications) from the image signal, it is necessary to return it to the state before the above conversion processing.

[0068] Fig. 9 is a flowchart showing an example of processing for calculating a cumulative count value from an image signal in embodiment 1. Note that the operation of each step in the flowchart in Fig. 9 is performed sequentially by a CPU or the like serving as a computer in the control and calculation unit 14 executing a computer program stored in memory. Below, the flow of processing for calculating a cumulative value of the number of avalanche amplifications (i.e., the signal value of the output signal) from an image signal will be described with reference to the flowchart in Fig. 9.

[0069] In step S900, an image is captured using a SPAD sensor. In step S901, the captured image is linearly converted and amplified (digital gain processing) with the conversion characteristics shown in Fig. 4(B) to obtain image signals for each predetermined region.

[0070] In step S902, the acquired image signal is inversely transformed after being amplified by the digital gain to calculate an image signal corresponding to the pulse count number. In step S903, a nonlinear back transformation is performed. That is, the image signal obtained by the inverse transformation in step S902 is further inversely transformed based on the nonlinear characteristics shown in FIG. 4(A) to convert it into an image corresponding to the number of input photons.

[0071] Next, in step S904, the number of pulses for each region is accumulated. That is, based on the image signal that has been nonlinearly back-converted in step S903, the signal processing unit 12 or the control and calculation unit 14 as a calculation unit calculates the number of avalanche amplification occurrences (output signals) in the image sensor 11 for each region. Here, step S904 functions as a calculation step (calculation unit) that calculates the number of avalanche amplification occurrences in the image sensor.

[0072] In step S904, the number of pulses while the camera is activated and continues to capture images is accumulated for each divided region, and this accumulated value is stored in memory 16 or an auxiliary storage device (not shown). Here, memory 16 or an auxiliary storage device (not shown) functions as a storage unit that stores the accumulated value of the number of occurrences for each region of the image signal, and step S904 functions as a storage step that stores the accumulated value of the number of occurrences for each region of the image signal.

[0073] In this manner, in this embodiment, the cumulative number of occurrences of avalanche amplification is calculated and stored for each divided region for the captured image, so that the influence of stress changes on the imaging element can be estimated.

[0074] Since the cumulative number can be a very large value depending on the brightness of the subject and the operation period of the imaging device, the cumulative number may be stored as a logarithm, stored in floating-point notation, or stored as an exponent part and an integer part.

[0075] Alternatively, instead of storing the number of pulses, it is possible to store, for example, the cumulative value of the luminance signal for each divided region obtained in step S901, or the cumulative value of the luminance signal for each divided region for the image signal obtained in step S903. Then, the stored cumulative value of the luminance signal for each divided region may be converted into the cumulative value of the number of occurrences of avalanche amplification at a desired timing.

[0076] Alternatively, instead of storing the cumulative value of, for example, the luminance signal for each divided region as described above, it is also possible to store the difference value or ratio value between, for example, the luminance signal for each divided region and a reference value (for example, Yref), or a value obtained by logarithmically converting the ratio value.Then, at a desired timing, these stored values ​​may be converted into the cumulative value of the number of occurrences of avalanche amplification, and the degree of stress change may be estimated based on the number of occurrences (number of pulses).

[0077] Furthermore, the Av, Tv, and Sv values, and the Ev value representing absolute brightness, may be accumulated and stored for each divided region. In this case, the ΔBv value based on the reference value is calculated by subtracting the accumulated Av, Tv, and Sv values ​​from the accumulated Ev value, so the number of pulses can be estimated using the reference value.

[0078] Furthermore, with regard to accumulating the number of pulses, in applications such as surveillance cameras that continuously capture images of a fixed subject for a long period of time, it is not necessary to accumulate the number of pulses for all consecutive frames; it is acceptable to calculate and accumulate the number of pulses for each area in the image at regular intervals.

[0079] 10 is a graph showing an example of the correlation between the number of avalanche amplifications and the amount of dark current noise generated per unit area according to embodiment 1. In FIG. 10, the horizontal axis represents the number of avalanche amplifications, but the number shown here is very large, on the order of mega or giga. The vertical axis represents the amount of dark current noise generated in an image due to degradation per unit area. An increase in the amount of noise can result in differences in brightness evaluation values ​​resulting from changes in exposure.

[0080] Fig. 11 is a diagram showing an example of changes in image brightness when exposure is controlled by aperture or exposure time in embodiment 1. The example in Fig. 11 shows an example of a brightness evaluation value (b-1) when there is little noise (a-1) and a brightness evaluation value (b-2) when there is a lot of noise (a-2). Before brightening using aperture or exposure time, the brightness evaluation values ​​(b-1) and (b-2) are the same.

[0081] However, as shown in Figure 11, when the aperture or exposure time is used to brighten the image, the brightness corresponding to the noise does not change due to the aperture or exposure time control, and only the actual brightness of the subject after subtracting the amount of noise increases. As a result, the brightness evaluation value, which was the same before the brightening, will deviate after the exposure is changed.

[0082] Fig. 12 is a diagram showing an example of changes in image brightness when exposure is controlled using digital gain in embodiment 1. Fig. 12 shows an example in which the same brightness evaluation values ​​(b-1) and (b-2) are obtained when there is little noise (a-1) and a lot of noise (a-2).

[0083] When digital gain is used to brighten an image, the image signal containing noise is digitally amplified, which also amplifies the noise. As a result, the image quality deteriorates due to the amplification of noise, but there is no discrepancy between the brightness evaluation values ​​(b-1) and (b-2).

[0084] In this way, the effect of noise increase due to the number of avalanche amplifications differs depending on the exposure parameters (aperture, exposure time, digital gain) controlled during automatic exposure adjustment. In other words, if the exposure is controlled to change significantly in accordance with exposure changes in aperture and exposure time, the exposure will change too much when controlled by digital gain, resulting in instability.

[0085] Conversely, if the exposure control amount is reduced in accordance with exposure changes due to digital gain, it may take a long time to reach the correct exposure in the aperture or gain range, or the correct exposure may stop short of being reached.

[0086] Therefore, in this embodiment, the cumulative value of the number of times avalanche amplification occurs is used. As shown in Figure 10, the cumulative value of the number of times avalanche amplification occurs is correlated with the change in dark current in each region of the SPAD sensor. Therefore, the cumulative value of avalanche amplification is obtained for each region.

[0087] 13 is a diagram showing an example of the cumulative count value of avalanche amplification by region according to embodiment 1. If the threshold at which the change in dark current starts to become noticeable is 150M (Mega), the reliability of the signal value is low in regions exceeding 150M.

[0088] Therefore, if the signal values ​​of this region are used primarily in the brightness evaluation process in step S502, the brightness evaluation results will not be stable. Therefore, in this embodiment, in step S502, the weight table used for the weighted average is changed according to the cumulative number of times for each region.

[0089] Fig. 14 is a diagram showing an example of the relationship between the cumulative count value for each region and the subtraction rate of the weighting coefficient when evaluating brightness in step S502. Fig. 14 shows a graph of the relationship between the cumulative value and the rate at which the weighting coefficient is reduced. If the cumulative value for each region exceeds 150M times, the value in the weighting table corresponding to that region is reduced, and if the cumulative value exceeds 300M times, the weight of that region is set to zero. In other words, weighting is performed so that the weight of the image signal for a region whose cumulative value exceeds a predetermined number of times is set to zero.

[0090] The weighting table obtained by correcting the weighting table shown in Fig. 7 is shown in Fig. 15. Fig. 15(A) to (C) are diagrams showing examples of corrected weighting tables when evaluating brightness in step S502.

[0091] 15, in step S502, the brightness is evaluated by weighting the image signals in areas with large cumulative values ​​so that the weights are reduced. In this way, in step S502, by correcting the weighting table according to the number of cumulative counts, the weights of areas with low reliability are reduced when evaluating the brightness, thereby making it possible to obtain stable brightness evaluation results.

[0092] In the example of Figure 14, an example was described in which the rate of decrease of the weighting coefficient increases linearly as the cumulative count increases from 150M to 300M, but it may also increase nonlinearly or according to a multidimensional function.

[0093] <Embodiment 2> In the second embodiment, brightness evaluation is performed using a method different from that of the first embodiment. Note that a description of parts that overlap with the first embodiment will be omitted.

[0094] When the Y signal (luminance signal) obtained by multiplying RGB by a predetermined coefficient and then adding them together is used as the signal used to evaluate brightness, the contribution of color signals with a low generation ratio is low. Therefore, when a primary color subject occupies a large proportion of the screen, the automatic exposure control can result in over-brightness.

[0095] In contrast, in this embodiment, the weighted average of RGB is not calculated, but the color signal used to evaluate the brightness is selected for each area. This prevents over-brightness even in primary color subjects, and allows for an evaluation of brightness that is closer to what the eye sees. This brightness evaluation method will be explained using the flowchart shown in FIG.

[0096] Fig. 16 is a flowchart showing an example of a process for evaluating brightness using a color signal according to embodiment 2. Note that the operations of the steps in the flowchart in Fig. 16 are performed sequentially by a CPU or the like serving as a computer in the control and calculation unit 14 executing a computer program stored in memory. Note that the flow in Fig. 16 starts when the imaging device 100 is started.

[0097] First, in step S1600, an image is captured by the image sensor 11. The captured image is transmitted to the signal processing unit 12, where various types of image processing are performed. Next, in step S1601, color signals of each region of the image that has been subjected to the image processing described above are acquired and integrated for each color signal.

[0098] Then, in step S1602, the color signal values ​​within the regions are compared and the largest one is selected. That is, the color signal with the largest signal value that is most easily reflected in human perception is selected. Next, in S1603, C is calculated by weighting and adding the color signals selected for each region. That is, the color signals selected for each region are multiplied by weights based on the weight table for each region, and the multiplication results for all regions are added to calculate the weighted average value C.

[0099] Then, in step S1604, the difference ΔC between Cref, which is a reference value of appropriate brightness stored in advance in the memory 16, and the weighted average value C is calculated using the following equation 2. ΔC=C-Cref (Formula 2)

[0100] If ΔC is smaller than a predetermined lower limit, the brightness is insufficient, and if ΔC is larger than a predetermined upper limit, the brightness is excessive. In other words, the brightness can be evaluated based on the color signal.

[0101] Furthermore, for example, if you continue to photograph a subject that has a high concentration of a specific color component, the number of avalanche amplifications will increase in the pixels that are sensitive to that color signal, which may lead to stress changes in the pixels of that specific color component, increasing noise and reducing reliability.

[0102] In such cases, when evaluating brightness using color signals, selecting color signals based solely on the magnitude of their values ​​may result in an inaccurate determination of the color signals of the subject. Therefore, in this embodiment, before selecting a color signal, the signal value of each color signal is corrected using the cumulative value of avalanche amplification.

[0103] FIG. 17 is a diagram showing an example of the relationship between the correction coefficients used to determine color signal selection in the second embodiment and the cumulative count values ​​by region, and shows an example of the correction coefficients for correcting the color signals used for the determination according to the cumulative number of times.

[0104] If the threshold at which changes in dark current begin to become noticeable is 150M (Mega), then the reliability of signal values ​​in areas exceeding 150M is low. Therefore, for example, signal values ​​with cumulative values ​​exceeding 150M are multiplied by a correction coefficient of 100% or less to make them less likely to be selected as color signals.

[0105] An example of signal values ​​in a certain region in an image and cumulative values ​​for that region will be described with reference to Fig. 18. Fig. 18(A) to (C) are diagrams showing examples of color signal values, cumulative values, and corrected color signal values ​​in a certain region according to the second embodiment.

[0106] In Figure 18(A), the signal values ​​of a certain area are shown by color. If a color signal is selected based on these signal values, the G signal will be selected as the signal value for this area because it is the largest.

[0107] On the other hand, Figure 18(B) shows the cumulative values ​​of each color component, and looking at these cumulative values, the R and G signals exceed 150 M, so an increase in dark current is expected for the R and G signals. In particular, the cumulative value of the G signal is the largest, so the reliability of the G signal is thought to be the lowest.

[0108] Figure 18(C) shows the results of correcting each color signal based on the correction coefficients in Figure 17. That is, in Figure 18(C), taking into account the decrease in reliability due to stress changes caused by the number of avalanche amplification occurrences, the maximum color signal is determined to be the B component, and the B component is selected as the signal value for this region.

[0109] In this way, steps S1602 to S1604 function as evaluation steps for evaluating the brightness of the image based on the cumulative value. Also, in steps S1602 to S1604, by selecting the color components of the region taking into consideration the cumulative value of the number of times avalanche amplification has occurred, it is possible to select color signals more accurately and evaluate the brightness even when avalanche amplification has progressed.

[0110] That is, in this embodiment, a predetermined color signal is selected for each region, and the brightness is evaluated based on the selected color signal. Also, the cumulative value of the number of avalanche amplification occurrences is the cumulative value for pixels of the same color included in each region, and a predetermined color signal is selected for each region for evaluating the brightness based on the cumulative value.

[0111] Step S1605 is a process for automatic exposure control, and controls at least one of the amount of light incident on the image sensor, the exposure time of the image sensor, and the gain of the image processing unit based on the brightness evaluation, similar to step S504 in Fig. 5. After step S1605, the process returns to step S1600 again, and the flow in Fig. 16 is repeated.

[0112] <Embodiment 3> In the third embodiment, a region of interest is set in an image, and the brightness of the entire image is evaluated by taking a weighted average of the signal values ​​of the region of interest and the signal values ​​of other regions.

[0113] That is, the present invention has a region-of-interest setting unit that can set a region of interest in an image, and evaluates the brightness of the image by taking a weighted average of the image signal of the region of interest and the image signal of other regions. The region of interest is set based on the cumulative number of occurrences of avalanche amplification. Explanations of parts that overlap with those of the first and second embodiments will be omitted.

[0114] 19(A) and 19(B) are diagrams showing an example of setting an attention area in embodiment 3. In Fig. 19, an image captured at an angle of view that includes a landscape 190 including the sun and a person area 191 is displayed on the operation display unit 17.

[0115] 19(A), the user can touch any area on the operation display unit 17 to set any area in the image as the attention area 192. The size of the settable area can be set in advance by the user.

[0116] Furthermore, once a region is set, it may remain fixed at the same position within the angle of view, or if a moving subject is present within the region of interest 192, the region of interest 192 may move within the angle of view to track the moving subject. In other words, the region of interest can be set and moved to any region within the image, starting from the region specified by the user.

[0117] Next, an example of automatically setting a region of interest in the imaging device without receiving settings from the user will be described with reference to Fig. 19(B). An important subject in an image may be detected by image recognition, and the imaging optical system 10 may be controlled to focus on the important subject or to prioritize and adjust the brightness of the important subject to an appropriate level.

[0118] At this time, a pre-trained object classifier is stored in the signal processing unit 12, and the object area is detected by image recognition based on the input image, and area information within the image of the detected object is notified to the control and calculation unit 14. The control and calculation unit 14 drives the focus lens of the imaging optical system 10 so that the contrast of the image in the notified area is maximized.

[0119] Furthermore, the subject area in the image is set as the area of ​​interest, and the brightness of the entire image is evaluated by taking the weighted average of the signal values ​​of the area of ​​interest and the other areas.Then, based on the evaluation result, the aperture in the imaging optical system 10, the exposure time of the image sensor 11, and the digital gain in the image signal processing shown in Figure 3 are controlled.

[0120] The position of the set attention area may be superimposed on the captured image in a display format such as a frame display, and the attention area may be shown to the user by being displayed on the operation display unit 17. Fig. 19(B) shows an example in which, when a human face area is detected as the important subject, the set attention area is presented to the user as a frame display via the operation display unit 17.

[0121] Since the area set as the region of interest is used preferentially for brightness evaluation, the brightness of the captured image is strongly influenced by the signal value of the region of interest. In this case, if the set region of interest includes an area where the number of avalanche amplifications is high, the brightness may be strongly influenced by noise due to stress changes, resulting in an unstable exposure state.

[0122] To address this issue, in this embodiment, unstable exposure is avoided by reducing the weight of the region of interest. For example, when the signal value of the region of interest is YROI, the signal value of the other regions is YOTHER, and the weight of the region of interest is α, the brightness evaluation value of the entire image is calculated using the following equation 3. Y=α×(YROI)+(1−α)×(YOTHER) (Formula 3)

[0123] In this case, if α is large, the signal value of the region of interest is given higher priority, so if the region of interest includes an area where the accumulated value of avalanche amplification is greater than a predetermined value, the effect of noise will be greater. Therefore, in this embodiment, the relationship between the maximum accumulated value of each region included in the region of interest and the weighting coefficient α is set, for example, as shown in Figure 20.

[0124] 20 is a diagram showing an example of the relationship between the cumulative count value by region and the weighting coefficient α according to the third embodiment. Here, the weighting coefficient α is set to a large value (e.g., 80%) while the cumulative value is below a predetermined value (e.g., 150M), and once the cumulative value exceeds a predetermined value (e.g., 150M) at which the influence of noise begins to become noticeable, the weighting coefficient α for the region of interest is decreased to reduce the influence of noise. This makes it possible to avoid unstable exposure when controlling the weight of brightness evaluation.

[0125] 21 shows a modified example of embodiment 3. Figures 21(A) to 21(C) are diagrams showing an example of resetting a region of interest according to embodiment 3. Figure 21(A) shows an example in which, when the set region includes a region where the accumulated value of avalanche amplification is larger than a predetermined value, the set region is changed to a larger set region 194 that includes a region where the accumulated value is smaller than the predetermined value, thereby making the region less susceptible to the effects of noise.

[0126] 21(B), when a set region of interest 192 includes a region 195 where the cumulative value of avalanche amplification is greater than a predetermined value, the set region is changed so as to exclude the region 195 where the cumulative value is greater than the predetermined value from the region of interest 192. Furthermore, the excluded region 195 is displayed, for example, in a predetermined color so that the user can recognize it.

[0127] 21(C) shows an example in which, if the set area includes a person area 191 where the cumulative value is greater than a predetermined value, the set area is canceled. That is, the area is excluded from the brightness evaluation target, and furthermore, a message that the set area is invalid is displayed to the user in a predetermined display format 196.

[0128] Note that the normal setting area is displayed with a frame of a predetermined color (e.g., green) at a constant brightness, and the display format 196 is displayed with a blinking frame or a red frame, etc. Furthermore, a warning such as "The specified area may not be AE stable. Please specify again" may be displayed.

[0129] In this way, even if the region set as the region of interest includes a region where the cumulative value of avalanche amplification is greater than a predetermined value, the size or range of the set region can be changed, the weighting coefficient can be changed, or a warning, etc. can be displayed, thereby making it possible to avoid an unstable exposure state.

[0130] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0131] The present invention also includes those that realize the functions of the above embodiments using, for example, at least one processor such as a CPU, memory, or circuit (for example, ASIC). Also, multiple processors may be used to perform distributed processing.

[0132] In order to realize some or all of the control in the above-described embodiments, a computer program that realizes the functions of the above-described embodiments may be supplied to an imaging device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the imaging device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. The present invention also includes the following combinations.

[0133] (Configuration 1) An imaging device comprising: an imaging element having an avalanche photodiode; an image processing unit that generates an image signal based on the output of the imaging element; a calculation unit that calculates the number of times avalanche amplification occurs in the imaging element; a memory unit that stores a cumulative value of the number of occurrences for each region of the image signal; and an evaluation unit that evaluates the brightness of an image based on the cumulative value.

[0134] (Configuration 2) The imaging device according to configuration 1, wherein the evaluation section evaluates the brightness by weighting the image signal for each region.

[0135] (Configuration 3) The imaging device according to configuration 2, wherein the evaluation section weights the image signal in the area where the cumulative value is large so that the weight of the image signal is small.

[0136] (Configuration 4) The imaging device according to configuration 3, wherein the evaluation section weights the image signal in an area where the cumulative value exceeds a predetermined number of times so that the weight of the image signal becomes zero.

[0137] (Configuration 5) The imaging device according to any one of configurations 1 to 4, wherein the calculation section calculates the number of occurrences of avalanche amplification in the imaging element based on the image signal.

[0138] (Configuration 6) An imaging device according to any one of configurations 1 to 5, characterized in that it has an exposure control unit that controls at least one of the amount of light incident on the imaging element, the exposure time of the imaging element, and the gain of the image processing unit based on the evaluation.

[0139] (Configuration 7) The imaging device according to any one of configurations 1 to 6, wherein the cumulative value is a cumulative value for pixels of the same color included in each of the regions.

[0140] (Configuration 8) The imaging device according to configuration 7, wherein the evaluation unit selects a predetermined color signal for each of the regions and evaluates the brightness based on the selected color signal.

[0141] (Configuration 9) The imaging device according to configuration 8, wherein the evaluation section selects the predetermined color signal for each of the regions based on the cumulative value.

[0142] (Configuration 10) An imaging device described in any one of configurations 1 to 9, characterized in that it has a focus area setting unit that can set a focus area in an image, and the evaluation unit evaluates the brightness by taking a weighted average of the image signal of the focus area and the image signal of other areas.

[0143] (Configuration 11) The imaging device according to configuration 10, wherein the attention area setting section sets the attention area based on the accumulated value.

[0144] (Method) An imaging method characterized by comprising an image processing step of generating an image signal based on the output of an imaging element having an avalanche photodiode; a calculation step of calculating the number of times avalanche amplification occurs in the imaging element; a storage step of storing a cumulative value of the number of occurrences for each region of the image signal; and an evaluation step of evaluating the brightness of the image based on the cumulative value.

[0145] (Program) A computer program for controlling each unit of the imaging device according to any one of configurations 1 to 11 by a computer. [Explanation of symbols]

[0146] 10: Imaging optical system 11: Image sensor section 12: Signal processing section 13: Video output section 14: Control and calculation unit 15: Memory 20: Photodiode 21: Quench resistance 22: Buffer 190: Landscape area including background 191: Person area 192: Area of ​​interest 193: Area of ​​interest including detected subject 194:Reset area of ​​interest 195: Invalid area in the set area 196: Display method for notifying cancellation of setting

Claims

1. an imaging element having an avalanche photodiode; an image processing unit that generates an image signal based on an output of the imaging element; a calculation unit that calculates the number of occurrences of avalanche amplification in the imaging element; a storage unit that stores the cumulative value of the number of occurrences for each region of the image signal; and an evaluation unit that evaluates the brightness of an image based on the cumulative value.

2. 2. The imaging device according to claim 1, wherein the evaluation unit evaluates the brightness by weighting the image signal for each region.

3. 3. The imaging device according to claim 2, wherein the evaluation unit weights the image signals in the area where the cumulative value is large so that the weight of the image signals in the area where the cumulative value is large is small.

4. The imaging device according to claim 3 , wherein the evaluation unit weights the image signal in an area where the cumulative value exceeds a predetermined number of times so that the weight of the image signal becomes zero.

5. The imaging device according to claim 1 , wherein the calculation unit calculates the number of occurrences of avalanche amplification in the imaging element based on the image signal.

6. 2. The imaging device according to claim 1, further comprising an exposure control unit that controls at least one of the amount of light incident on the imaging element, the exposure time of the imaging element, and the gain of the image processing unit based on the evaluation.

7. The imaging device according to claim 1 , wherein the cumulative value is a cumulative value for pixels of the same color included in each of the regions.

8. 8. The imaging device according to claim 7, wherein the evaluation unit selects a predetermined color signal for each of the regions, and evaluates the brightness based on the selected color signal.

9. 9. The imaging device according to claim 8, wherein the evaluation section selects the predetermined color signal for each of the regions based on the cumulative value.

10. 2. The imaging device according to claim 1, further comprising an attention area setting unit capable of setting an attention area in an image, wherein the evaluation unit evaluates the brightness by a weighted average of an image signal of the attention area and an image signal of other areas.

11. The imaging device according to claim 10 , wherein the attention area setting unit sets the attention area based on the cumulative value.

12. an image processing step for generating an image signal based on an output of an imaging element having an avalanche photodiode; a calculation step of calculating the number of occurrences of avalanche amplification in the imaging element; a storage step of storing the cumulative value of the number of occurrences for each region of the image signal; and an evaluation step of evaluating the brightness of the image based on the cumulative value.

13. A computer program for controlling each unit of the imaging device according to any one of claims 1 to 11 by a computer.

Images