Image pickup apparatus, control method, storage medium, and program product

Abstract

The invention provides an imaging apparatus, a control method, a storage medium, and a program product. The image pickup apparatus includes: an image pickup element including an avalanche photodiode configured to photoelectrically convert an optical image; and at least one processor and a memory coupled to the at least one processor, the memory storing instructions that, when executed by the at least one processor, cause the at least one processor to: generate an image signal based on an output signal from the imaging element; calculating the number of occurrences of avalanche amplification in the imaging element based on the image signal; storing the accumulated value of the occurrence frequency; and controlling a parameter related to exposure in a case where it is determined that the brightness of the image based on the image signal is outside a predetermined range determined based on the integrated value.

Description

Technical Field

[0001] The present disclosure relates to an imaging device, a method and a storage medium. Background Art

[0002] A single photon avalanche diode (SPAD) sensor (hereinafter referred to as a SPAD sensor) is proposed as an image sensor. In a SPAD sensor, an avalanche amplification phenomenon is used, in which electrons are accelerated when a strong electric field is applied and the electrons collide with other electrons to arouse multiple electrons, and an avalanche-like phenomenon is caused to generate a large current. Accordingly, weak photons input to a pixel can be converted into a large current and detected as an electric charge. Due to the mechanism of the SPAD sensor, since no noise is generated when the signal is read out, the SPAD sensor is expected to be applied as an image sensor. In particular, since the SPAD sensor can clearly capture a subject without being affected by noise even in a dark place, it is expected that the SPAD sensor is widely used as an image sensor for monitoring purposes, etc.

[0003] When considering surveillance purposes, since the SPAD sensor is set at a fixed position to continuously capture action for a long time, it is preferable to automatically adjust the exposure according to the brightness of the subject. In addition, C. Zhang, "SPAD requirements from consumer electronics to automotive", Int. SPAD Sensor Workshop (2022) reported the following phenomenon: When a photoelectric conversion device having an avalanche photodiode (APD) is driven for a long time, the amount of dark current changes. Summary of the invention

[0004] According to aspects of the present invention, there is provided an imaging device, comprising: an imaging element, comprising an avalanche photodiode configured to perform photoelectric conversion on an optical image; and at least one processor and a memory coupled to the at least one processor, the memory storing instructions which, when executed by the at least one processor, cause the at least one processor to: generate an image signal based on an output signal from the imaging element; calculate the number of occurrences of avalanche amplification in the imaging element based on the image signal; store a cumulative value of the number of occurrences; and, in a case where it is determined that the brightness of an image based on the image signal is outside a predetermined range, control parameters related to exposure, wherein the predetermined range is determined based on the cumulative value.

[0005] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram illustrating the internal structure of the imaging apparatus.

[0007] Figure 2 This is an equivalent circuit diagram of a pixel constituting an image pickup element.

[0008] Figure 3 The flow of generating an image signal is illustrated below.

[0009] Figure 4A is a graph showing the relationship between the number of pulse counts and the number of photons, and Figure 4B It is a diagram showing the relationship between the image signal and the number of pulse counts.

[0010] Figure 5 It is a flow chart of the automatic exposure control function.

[0011] Fig. 6A illustrates an example of a sensed image, and Figure 6B An example of a signal value of a sensed image is illustrated.

[0012] Fig. 7A , Figure 7B and Figure 7C An example of a weight table for evaluating lightness is illustrated.

[0013] Figure 8 An example of an automatic exposure (AE) line graph is illustrated.

[0014] Fig. 9 It is a flow chart for calculating the cumulative count value.

[0015] Fig.10 This is a graph showing the relationship between the number of occurrences of avalanche amplification and the amount of noise generated per unit area.

[0016] Fig.11 An example of image brightness change in exposure control based on aperture or exposure time is illustrated.

[0017] Fig.12 An example of image brightness change in exposure control based on digital gain is illustrated.

[0018] Fig.13 is a flow chart for updating exposure parameters based on the number of occurrences of avalanche amplification by region.

[0019] Fig.14 An example of cumulative count values ​​by region is illustrated.

[0020] Fig.15 The relationship between the accumulated count value for each area and the exposure response coefficient is illustrated.

[0021] Fig.16The relationship between the cumulative count value for each area and the threshold value for proper exposure determination is exemplified. DETAILED DESCRIPTION

[0022] Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. The embodiments described below are examples of devices configured to implement the present invention. The embodiments will be appropriately modified or changed according to the device structure and various conditions to which the present invention is applied, and the present invention is not limited to the following examples.

[0023] First embodiment

[0024] Equipment structure

[0025] Figure 1 1 is a block diagram illustrating the structure of the imaging apparatus according to the present embodiment. The imaging optical system 10 includes a zoom lens, a focus lens, an image stabilization lens, an aperture, a neutral density (ND) filter, etc. Each unit constituting the imaging optical system 10 is controlled to determine the imaging angle, focus, zoom, exposure, etc. when a subject is to be photographed. The imaging optical system 10 forms an optical image of the subject on the imaging element 11.

[0026] The imaging element 11 is an imaging element configured to perform photoelectric conversion for converting an optical image formed in each pixel on the image sensing surface into an electrical signal. Each pixel constituting the imaging element 11 is an avalanche photodiode configured to perform photoelectric conversion on an optical image, and serves as a SPAD sensor configured to count the number of photons forming the optical image. An output signal from the imaging element 11 is output to the signal processing unit 12 to apply various types of image processing by the signal processing unit 12.

[0027] The signal processing unit 12 is an image processing unit configured to generate an image (image signal) based on the output signal from the imaging element 11. The signal processing unit 12 is an image processing engine configured to perform correction processing such as the removal of fixed pattern noise included in the output signal from the imaging element 11, brightness correction based on digital gain, demosaic processing, edge enhancement processing, and gamma processing. The signal processing unit 12 is not used only for the above-mentioned correction processing. The recognition processing of detecting the subject area from the image (image signal) to control the imaging optical system 10 is also performed by the signal processing unit 12. In addition, the generation of evaluation values ​​used for exposure control or white balance (WB) correction is also performed by the signal processing unit 12, and the generated evaluation values ​​are sent to the control operation unit 14. The specific processing contents inside the imaging element 11 and the signal processing unit 12 will be described in detail below. The image subjected to the correction processing by the signal processing unit 12 is sent to the video output unit 13.

[0028] The video output unit 13 outputs the image subjected to the correction processing to an external device outside the camera connected via an output terminal. The video output unit 13 receives an image size and a video frame rate at which the external device can receive the image, and outputs an image synchronization signal together with the output image. Here, any terminal can be used as the output terminal as long as the video signal can be exchanged with the external device. A serial digital interface (SDI), a high-definition multimedia interface (HDMI) (registered trademark), and a universal serial bus (USB) can be used, and further a terminal such as a standard socket (RJ)-45 or the like can be used. In addition, a terminal that conforms to a unique standard can be used, with which a video signal can be exchanged with a specific external device.

[0029] The control operation unit 14 generates control information to be sent to the imaging optical system 10, the imaging element 11, the signal processing unit 12, and the video output unit 13. The control operation unit 14 includes a central processing unit (CPU) 15, and generates control information when the CPU 15 executes control processing stored in the memory 16 or an auxiliary storage device not illustrated in the figure.

[0030] The memory 16 is an area for storing data for performing operations of control information. Data in the middle of the operation or the operation result is stored in the memory 16, and the CPU 15 performs the operation in the control operation unit 14 while appropriately referring to the memory 16. The image signal generated by the signal processing unit 12 can be stored in the memory 16 through the control operation unit 14. Here, the memory 16 is illustrated as a block diagram inside the camera, but it can have a structure of a storage device that can be installed inside the camera and removed from the inside of the camera.

[0031] Image sensor structure

[0032] Next, refer to Figure 2The pixels of the SPAD sensor in the imaging element 11 are explained by using an equivalent circuit diagram of a pixel in. Each pixel is composed of a photodiode 20, a quenching resistor 21 and a buffer 22, and the photodiode 20 mentioned here is a SPAD. When a reverse voltage exceeding the breakdown voltage is applied, the photodiode 20 operates in Geiger mode by regarding the entry of a photon as a trigger. For this reason, a reverse bias voltage is used, which is equal to or higher than the breakdown voltage of the voltage VH applied to the SPAD. When a photon is input to the photodiode 20, an avalanche current is generated. The quenching voltage VQ applied to the quenching resistor 21 is set to be variable, and when the reverse bias voltage decreases based on the quenching resistor value relative to the avalanche current generated in the photodiode 20, the end of the avalanche amplification can be controlled. Here, as a specific structure of the quenching resistor 21, a metal oxide silicon field effect transistor (MOSFET) can be used, or other semiconductor elements with a quenching function can also be used. A threshold voltage is set in the buffer 22, and the buffer 22 is responsible for the shaping function of the waveform to output the H level in a specific time period due to the fluctuation of the input voltage. For the signal shaped by the buffer 22 in this way, a pulse waveform is output as a signal. For this pulse signal, the number of pulse waveforms is counted using a counter circuit prepared in the subsequent stage, and the number of pulses in a specific time period is output as a signal value of the pixel. Here, an explanation is provided based on a simple structure including a photodiode 20, a quenching resistor 21, and a buffer 22 as a circuit structure of each pixel in the SPAD sensor, but a pixel having other circuit elements or semiconductors can also be used as a circuit structure of the pixel.

[0033] Conversion from output signal to image signal

[0034] Next, refer to Figure 3 The process of processing the output signal of the pixel as an image signal is described below. First, the number of pulses counted for each pixel (corresponding to the number of occurrences of avalanche amplification) is output as the output signal of each pixel. However, it is noted that since there is a limitation in the time resolution of counting photons in the SPAD sensor, when the number of photons per unit time is very high, the photons can be counted without separating the photons from each other. As a result, the correlation between the number of pulses and the number of photons has the following relationship: Figure 4A The nonlinear characteristic is shown here as a probability function of the number of pulses. There is a nonlinear correlation between the number of photons and the number of pulses, and when the observed number of pulses is transformed by the inverse function of the nonlinear characteristic, the linear number of photons can be calculated. Figure 4BThe inverse conversion characteristics used to convert the number of pulses into the number of photons are illustrated. Therefore, when the signal processing unit 12 converts the number of pulses (output signal) by using the inverse characteristics of the probability function, an image signal that is linear with respect to the number of photons can be obtained. Next, as a device configured to control exposure, the brightness of the image is adjusted by applying amplification based on digital gain to the image signal. Therefore, an image signal that changes linearly with respect to the number of photons and is further maintained at an appropriate brightness can be obtained. Figure 3 , linear transformation and digital gain are mentioned as processes applied to the number of pulses for each pixel, but processes other than these processes may also be included. For example, the processing may include fixed pattern noise (FPN) correction for detecting and correcting defective pixels, uneven shading correction for correcting the black level within a plane, and optical correction for correcting degradation due to the characteristics of an optical lens, etc. In addition, the processing may include white balance processing for adjusting the white balance (WB) of an image, contour enhancement processing, and noise reduction processing. The above-mentioned image processing may be performed by the signal processing unit 12, and some of the processing may be performed in the imaging element 11.

[0035] Automatic exposure control function

[0036] Next, refer to Figure 5 The automatic exposure control function in the control operation unit 14 is explained with reference to the flowchart of FIG. 1. First, an image is sensed in the imaging element 11 (S500). The sensed image (output signal) is sent to the signal processing unit 12 to apply various types of image processing by the signal processing unit 12. Fig. 6A An example of an image subjected to signal processing is illustrated. In this image, a forest, a backlit building, the sky, and the sun are sensed. The sun has a very high signal value of the image signal, and then the signal value of the image signal further decreases in the order of description of the sky, the forest, and the backlit building. At this time, the control operation unit 14 divides the image into a plurality of areas, and obtains a signal value for each area to evaluate the brightness. The image is divided into four in the horizontal direction and four in the vertical direction, and Figure 6BSignal values ​​by region of an image signal are illustrated. Here, the signal value obtained from the image region reflecting the sun is very high, and the signal value obtained from the image region reflecting the backlit building is low. The signal values ​​mentioned here can be obtained for each color of a Bayer image, or can also be obtained for each color or each brightness of a YCC or YUV image. The number of divisions for the signal value by region does not necessarily need to be 4×4, and the signal value can be obtained by 8×8 or 16×16 regions. Here, an example of obtaining the average value of the signal value of the image signal by region is illustrated, but the signal value can be obtained as the sum of the signal values ​​in the region. In this way, signal values ​​are obtained by region in the image (S501).

[0037] Next, the signal values ​​obtained for each area are weighted to evaluate the brightness Y of the object ( S502 ). 7A to 7C An example of a weight table used when weighting each area to evaluate brightness is illustrated. Fig. 7A The weight table used when evaluating brightness at the same rate for all regions without changing the weight for each region is exemplified. Figure 7B A weight table used when evaluating brightness by considering a tendency that a high-brightness subject such as the sun is reflected in the upper part of an image and the weight in the area in the upper part of the image decreases (is set differently) is exemplified. Figure 7C The following weight table is illustrated: the weight in the central part of the image is increased by taking into account the tendency that the main subject may be reflected in the central part of the image, so that it is easy to accurately grasp the brightness of the main subject. In other words, the weight in the central part of the image is set to be different from the weight in the peripheral part of the image. Here, the type of fixed weight table for changing the weight for each area to evaluate the brightness is introduced, but these tables can be dynamically switched to evaluate the brightness according to the subject or shooting conditions. In addition, weight tables other than the above can be dynamically generated according to the type or position of the subject in the image to evaluate the brightness.

[0038] The signal value indicating the brightness of the image can be calculated based on the image signal value of each area obtained from the image as described above and the weight table. Specifically, the signal value of the image signal in each area is multiplied by the weight value in each area specified by the weight table, and the multiplication results in each area are added across all areas, so that the brightness Y of the object can be evaluated (calculated) (S502). The brightness Y obtained by calculation across all areas is compared with the reference value Yref set to the appropriate brightness that is pre-stored in the memory 16, so that the brightness of the object can be evaluated. That is, when Y is lower than the reference value Yref, it can be evaluated that the brightness of the object is insufficient, and when Y is greater than the reference value Yref, it can be evaluated that the brightness is too large.

[0039] In view of the above, ΔY is calculated as the difference between Y and Yref (S503). At this time, the brightness of the image does not need to be completely consistent with the reference value, and when the difference ΔY is lower than the predetermined threshold value θBv, it is judged that the brightness is within the range of appropriate brightness (within the predetermined range). Therefore, when the following expression 1 is satisfied, it is judged that Y is the appropriate brightness (S504).

[0040] ΔY = abs (Y - Yref) < θBv ··· (1)

[0041] Note that Yref is brightness set as a reference, and may be, for example, a predetermined percentage of the sum of the signal values ​​of the image signal. Alternatively, Yref may be a maximum predetermined percentage (eg, 18%) of the signal value of the image signal.

[0042] Next, the automatic exposure adjustment function (S505) applied when the brightness deviates from the appropriate brightness ("No" in S504) will be described. When the brightness deviates from the appropriate brightness, the brightness can be returned to the range of the appropriate brightness by adjusting the exposure-related parameters by the deviation amount ΔY in the direction in which the brightness is offset. That is, in the case where it is determined that the brightness Y of the image based on the image signal is outside the predetermined range, the exposure-related parameters are controlled. The exposure-related parameters include aperture, exposure time period, digital gain, etc., and which parameter to adjust is specified by the automatic exposure (AE) line diagram. Figure 8 An example of an AE line graph is illustrated. In this AE line graph, when the subject moves from the dark side to the bright side, first, the digital gain is set to the minimum to darken, and then the aperture is used to darken. After that, the exposure time period is adjusted to darken. Halfway through, a single-density ND filter is inserted, and the exposure time period is set to brighten due to the amount of darkening caused by the ND filter. Finally, the exposure time period is set to further darken. In this way, a control sequence is specified based on parameters related to exposure, and it becomes possible to appropriately control the brightness of the subject by combining parameters related to exposure. Here, as an example, the following case will be considered: when the brightness of the subject is Y1, the evaluation result evaluated as the brightness is brighter by ΔY, and it is judged that ΔY is higher than a predetermined threshold value θBv. Figure 8, at Y1, the brightness is at a level where the exposure time period becomes shifted. When the exposure time period shifts and becomes darker by ΔY, the brightness shifts to Y2. Here, since the exposure is shifted by ΔY, ΔY=0, so that the brightness falls within the appropriate brightness and the automatic exposure adjustment is stopped. Note that ΔY does not necessarily need to be set to 0 completely, and the parameters related to the exposure can be controlled so that ΔY is set below a predetermined threshold value θBv. In this way, the brightness is evaluated based on the image signals in each area of ​​the image, and by mutually controlling the exposure-related parameters specified in the AE line diagram, the subject can be controlled to be set to the appropriate brightness. Note that the function of a control unit configured to control the above-mentioned exposure-related parameters can be implemented by the control operation unit 14.

[0043] Stress changes in imaging elements

[0044] As a characteristic of the SPAD sensor, a large current flows and the number of photons can be counted due to the avalanche amplification phenomenon according to the input photons, which is described above. On the other hand, in order to generate the avalanche amplification phenomenon, a reverse bias voltage exceeding the breakdown voltage is applied, and when a large voltage is applied, a large current flows. Assuming the use as an image sensor, a shooting action for an image at more than 30 frames per second is performed in video shooting, and a large current repeatedly flows through the circuit elements of each pixel, making the load very large. Therefore, due to the repetition of the large current, the stress of the circuit element may change. When stress changes in the circuit elements of each pixel in the SPAD sensor, the dark count rate (DCR) may increase. In view of the above, when the number of occurrences of avalanche amplification in each pixel in the SPAD sensor is cumulatively counted, the DCR generated in each pixel can be roughly predicted. However, when the cumulative value of the number of occurrences of avalanche amplification is kept in the sensor in units of pixels, it is difficult to continue to keep the cumulative value in the counter circuit of the sensor because the amount of data to be stored further increases as the number of pixels on the sensor increases.

[0045] Calculation of output signals and storage of accumulated values

[0046] In view of the above, the case where the signal value of the output signal is obtained by region from the sensed image to be accumulated will be considered. The output signal indicates the number of occurrences of avalanche amplification (i.e., the number of pulses). According to the present embodiment, the output signal and the image signal are distinguished as different signals. The output signal is a signal output from the imaging element 11, and the image signal is a signal subjected to transformation processing by the signal processing unit 12. The signal value of the output signal can be obtained for each color of the Bayer image, or can also be obtained for each color or each brightness of the YCC or YUV image. The number of divisions for the signal value by region does not necessarily need to be 4×4, and the signal value can be obtained by 8×8 or 16×16 regions. An example of obtaining the signal value of the image signal by region based on the automatic exposure adjustment function is illustrated, but the signal value of the image signal by region that is the same as the case of automatic exposure adjustment can also be used. In addition, an example of obtaining the average value of the signal value of the output signal by region is illustrated here, but the signal value can also be obtained as the sum of the signal values ​​of the output signal in the region.

[0047] As described above, the obtained image signal by region is a signal obtained by applying some conversion processing to the output signal (pulse number). Therefore, the signal value of the image signal is different from the number of occurrences of avalanche amplification.

[0048] exist Figure 3 In the process shown, digital gain and linear transformation are applied to the output signal, but in order to calculate the output signal from the image signal, the applied transformation will be undone. Fig. 9 The flowchart of FIG. 1 is used to explain the processing flow of calculating the cumulative value of the number of occurrences of avalanche amplification (that is, the signal value of the output signal) from the image signal.

[0049] First, an image is sensed by using a SPAD sensor (S900). Linear transformation and digital gain are applied to the sensed image (output signal) to obtain an image signal by region (S901). Next, an amplification amount based on the digital gain is inversely transformed for the obtained image signal (S902). Figure 4BThe inverse transformed image signal is inversely transformed (S903) based on the nonlinear characteristics between the number of pulses in the SPAD sensor and the image signal. Note that there may be a case where the output signal from the SPAD sensor is not equivalent to the number of occurrences of avalanche amplification. For example, it is entirely conceivable that a signal offset relative to the number of pulses in the sensor is output as the output signal. In this case, in addition to the above-mentioned two inverse transformation processes, the signal processing unit 12 can also obtain the number of occurrences of avalanche amplification more accurately by performing a subtraction process of the signal value according to the offset in the sensor. In this way, the number of occurrences of avalanche amplification can be calculated by performing a predetermined transformation including at least one of the inverse transformation and the subtraction process on the image signal. However, note that the offset in the sensor can be allowed to regard the output signal as the number of occurrences of avalanche amplification as it is. In this case, the processing in S902 and S903 can be skipped by storing the output signal before the transformation process by the signal processing unit 12 as a cumulative value as it is. Note that the signal processing unit 12 or the control operation unit 14 implements a function as a calculation unit configured to calculate the number of occurrences of avalanche amplification in the imaging element 11 based on the image signal. During the period when the camera is activated to continue the image capturing operation, the number of occurrences of avalanche amplification (output signal, number of pulses) obtained by inverse transformation (calculation) is accumulated for each region, and the number of occurrences is stored in the memory 16 serving as a storage unit (S904). That is, the accumulated value of the number of occurrences of avalanche amplification calculated by the calculation unit (signal processing unit 12 or control operation unit 14) is stored in the storage unit (memory 16).

[0050] The memory 16 or an auxiliary storage device not illustrated in the figure functions as a storage unit configured to store the cumulative value of the number of occurrences of avalanche amplification calculated by the calculation unit.

[0051] By calculating and storing the number of occurrences of avalanche amplification using a calculation unit for each sensed image, the number of occurrences of avalanche amplification can be grasped for each region, and the influence of stress changes from circuit elements (increase in DCR) can be estimated.

[0052] Here, since the cumulative value may become an astronomical number depending on the brightness of the subject of the camera or the operation time period, as a method of maintaining the cumulative number of occurrences, the cumulative value may be stored in a logarithmic expression, or a floating point representation may be used. The cumulative value may be divided into an exponential part and an integer part for storage. The value stored as the cumulative value does not necessarily need to be the number of pulses itself, and may be other physical quantities by which the number of pulses can be calculated. For example, the cumulative value of the image signal for each area may be stored in the memory 16, and at the timing of referring to the stored cumulative value, the cumulative value of the image signal may be converted into the cumulative value of the output signal.

[0053] Alternatively, the accumulated value may be stored as a difference or ratio value with a value set as a reference (e.g., Yref, etc.), or may be stored as a value obtained by logarithmically transforming the ratio value. In this case, conversion to the number of pulses is performed by using these physical quantities and the value set as a reference, and the degree of stress change can be estimated based on the number of pulses.

[0054] The Ev value representing the absolute brightness taking into account the Av value, Tv value, and Sv value in the signal value of the image can be maintained for each region. In this case, since the ΔBv value based on the reference value is calculated by subtracting the accumulated Av value, Tv value, and Sv value from the accumulated Ev value, it becomes possible to estimate the number of pulses by using the reference value.

[0055] Regarding the accumulation of the number of pulses, it is not necessarily necessary to accumulate the number of pulses in all consecutive frames in applications such as surveillance cameras where the shooting action of a fixed subject continues for a long period of time, and the number of pulses can be calculated and accumulated by area in the image at intervals of a certain time period.

[0056] Exposure control based on cumulative value

[0057] Fig.10 The correlation between the number of avalanche amplification occurrences and DCR is shown in the figure. Fig.10 In FIG. 1 , the horizontal axis represents the number of occurrences of avalanche amplification. Here, the number of occurrences is very large and is assumed to be in the order of mega or giga. The vertical axis represents the amount of noise caused by dark current per unit area generated in the image due to degradation. When the amount of noise increases, a difference may occur in the brightness evaluation based on the change in exposure. Fig.11 Two examples are illustrated in which the same brightness evaluation result is obtained in the case where the amount of noise is large ((a)-1) and the case where the amount of noise is small ((b)-1). Here, when considering the case where the brightness is increased based on the aperture or exposure time period, since the brightness does not change the amount of noise in the control based on the aperture or exposure time period, the brightness is increased only by the true brightness of the subject obtained by subtracting the amount of noise. For this reason, in the two examples with the same brightness before the brightness increase, as Fig.11 As illustrated in (a)-2 and (b)-2, it was found that the two examples did not obtain the same brightness after the change in exposure.

[0058] Next, consider an example of changing the exposure based on digital gain. Fig.12Two examples are illustrated in which the same brightness evaluation result is obtained in the case where the amount of noise is large ((a)-1) and the case where the amount of noise is small ((b)-1). When the brightness is increased based on the digital gain, since amplification is digitally applied to the image signal including noise, the noise is also amplified. As a result, Fig.12 As illustrated in (a)-2 and (b)-2 in , since the noise is also amplified, the image quality deteriorates, but these two examples have the same result as the brightness evaluation result. In this way, regarding the influence of the degradation from the number of occurrences based on avalanche amplification, different phenomena also occur depending on the exposure-related parameters controlled during automatic exposure adjustment. That is, when control is performed to shift the exposure significantly based on exposure changes based on the aperture or exposure time period, the exposure shifts too much in the exposure control based on the digital gain, and a phenomenon occurs in which the exposure becomes unstable. In contrast, when control is performed to reduce the exposure control amount based on the exposure change based on the digital gain, the following phenomenon occurs: it takes time to reach the appropriate exposure in the aperture or gain area, or the exposure stops when the appropriate exposure is about to be reached.

[0059] In view of the above, it will be considered to use the cumulative value of the number of occurrences of avalanche amplification (hereinafter referred to as the cumulative value). Fig.13 A flowchart is illustrated when the integrated value is used for exposure control. It is found that the integrated value has a correlation with the change in dark current in each area of ​​the SPAD sensor. In view of the above, the integrated value is obtained for each area by referring to the storage unit (memory 16) (S1300). Fig.14 An example of cumulative values ​​by region is illustrated. Note that M indicates 10^6. When the threshold value T at which the change in dark current begins to become noticeable is 150M, eight regions out of 16 regions have a value exceeding 150M, and the increase in dark current is noticeable in 50% of the regions. In this way, as an example, the percentage of regions where the cumulative value becomes equal to or higher than the threshold value T at which the change becomes noticeable is calculated (S1301).

[0060] A description is provided of a case where the brightness may not change as much as the changed exposure amount due to the influence from noise even when the exposure is changed. Here, a method of achieving stable exposure control by using the percentage of areas where the cumulative value becomes equal to or higher than the threshold value T will be described.

[0061] First, in the control based on the aperture and the exposure time period, since the change in the amount of light is reduced compared to the control based on the digital gain, it takes time to make the automatic exposure adjustment. In view of the above, the following configuration is considered: by increasing the exposure response coefficient, the time period taken for the automatic exposure adjustment is returned to the original setting. The exposure response coefficient is a coefficient that specifies the duration of the unit time to reach the target exposure difference. How much the exposure is to be shifted during one frame is determined based on this coefficient. Fig.15 : is a relationship diagram illustrating the relationship between the percentage of the area where the integrated value becomes equal to or higher than the threshold value T and the exposure response coefficient.

[0062] 0.5 [EV / sec] is set when the percentage of the area where the integrated value becomes equal to or higher than the threshold value T is small. This indicates that a response of 0.5 seconds is required to change the exposure by 1 level in the exposure setting.

[0063] On the other hand, as the percentage of the area where the cumulative value becomes equal to or higher than the threshold value T becomes larger, the light amount change relative to the changed exposure decreases. Therefore, as the exposure response coefficient increases, the light amount change can increase. Fig.14 As shown in the example, it is assumed that the percentage of the area with obvious degradation is 50%. Fig.15 The exposure response coefficient is found to be 0.65 in the relationship diagram. In this way, the exposure response coefficient is determined based on the cumulative value of the number of occurrences of avalanche amplification (S1302). Accordingly, the phenomenon that it takes time to stop the exposure is avoided.

[0064] On the other hand, when the exposure amount of one shift increases, since the exposure amount shifted in one frame increases, a phenomenon may occur in which the exposure exceeds the range of appropriate exposure and the exposure does not converge. Fig.10 The increase in dark current is illustrated in , but the change does not necessarily occur in this way. That is, the stress change for the input light amount occurs together with the number of occurrences of avalanche amplification, but the degree of the stress change may vary. As a result, various possibilities are considered regarding the change in the signal value on the image for the exposure change amount, wherein the exposure response may return to the normal exposure response, and it is also conceivable that the exposure response shifts too quickly. In view of the above, the threshold value θBv used in the judgment of proper exposure is changed in the expansion direction. In other words, the range in which the brightness is judged to be proper brightness (the range in which the parameters related to the exposure will not change) is determined based on the cumulative value of the number of occurrences of avalanche amplification. Specifically, the value of the above-mentioned threshold value θBv is increased (the range in which the brightness is judged to be proper brightness is expanded). In addition, as shown from Fig.16It is deduced that, as the percentage (or the number of regions) of regions where the cumulative value becomes equal to or higher than the threshold value T increases, the value of θBv further increases (the range in which the brightness is judged to be the appropriate brightness expands). Accordingly, even when the exposure amount shifted in one frame increases, the brightness can fall within the appropriate range, and it becomes possible to stably stop the automatic exposure adjustment. Fig.16 The relationship between the percentage of the area where the cumulative value becomes equal to or higher than the threshold value T and the threshold value (range) for proper exposure judgment is illustrated. When the percentage is low, θBv=0.3 [level]. However, in order to avoid unstable exposure as a result of an increase in response, when the threshold value is updated to θBv=0.45 [level] (S1303), the shooting action can be performed with stable exposure. In this way, by changing the exposure response coefficient, or the threshold value (range) for proper exposure judgment according to the degree of degradation, the exposure can be stably controlled even when the change in the characteristics of the SPAD sensor progresses.

[0065] In the above description, the control of exposure using the percentage of the area where the number of occurrences of avalanche amplification is equal to or higher than the threshold is described, but other indicators can be used. For example, the maximum value of the cumulative value can be used, or the cumulative value of the number of occurrences of avalanche amplification in a specific area such as the central part of the image can also be used. In other words, the exposure response coefficient or the threshold value (range) of the appropriate exposure judgment is determined based on the maximum value of the cumulative value of the number of occurrences of avalanche amplification. Alternatively, the exposure response coefficient or the threshold value (range) of the appropriate exposure judgment is determined based on the cumulative value of the number of occurrences of avalanche amplification in a specific area (for example, the central part or a specified area). The cumulative value of the number of occurrences of avalanche amplification can be stored for each pixel, and the exposure response coefficient or the threshold value (range) of the appropriate exposure judgment can be determined based on the area ratio of the pixels whose cumulative value becomes equal to or higher than the threshold value T relative to the entire area or the number of these pixels. The cumulative value can be reset due to reasons such as replacement of the sensor. In this case, the exposure response coefficient can be returned to the original setting, or the threshold value (range) of the appropriate exposure judgment can also be reduced.

[0066] Note that, according to the present embodiment, the case where both the exposure response coefficient and the threshold value (range) for proper exposure determination are changed based on the cumulative value of the number of occurrences of avalanche amplification is described, but only one of the two may be changed. Fig.13 In the flowchart of , S1302 can be skipped and only S1303 can be performed. With the processing in only S1303, the threshold θBv is changed so that the brightness Y does not erroneously exceed the threshold for proper exposure judgment due to an increase in dark current, and the advantage of stabilizing AE can be obtained.

[0067] This also applies to the relationship between the exposure response coefficient and the percentage of the area where the cumulative value becomes equal to or higher than the threshold value T. Fig.15 In , a linear relationship is described as an example, but a nonlinear correspondence can be used, or a multidimensional function can also be used. This also applies to Fig.16 The relationship between the percentage of the area where the cumulative value becomes equal to or higher than the threshold value T and the threshold value (range) for proper exposure judgment is illustrated, but a relationship other than this relationship may be used. This figure only illustrates an example indicating the association with the percentage of the area where the cumulative value becomes equal to or higher than the threshold value T. Therefore, this figure does not limit any relationship (the relationship between the percentage of the area where the cumulative value becomes equal to or higher than the threshold value T and the exposure response, or the relationship between the percentage of the area where the cumulative value becomes equal to or higher than the threshold value T and the threshold value (range) for proper exposure judgment).

[0068] Other embodiments

[0069] The present invention has been described in detail above by way of the embodiments, but the present invention is not limited to these specific embodiments, and various modes within the scope not departing from the gist of the present invention are also included in the present invention. Some of the above embodiments may be appropriately combined.

[0070] The present invention can also be implemented by supplying a program for implementing one or more functions of the above-described embodiments to a system or device via a network or storage medium, and one or more processors in a computer of the system or device read out and execute the program.

[0071] Additionally, the present invention may be implemented by a circuit (eg, an application specific integrated circuit (ASIC)) configured to implement one or more functions.

[0072] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0073] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications, equivalent structures and functions.

Claims

1. A camera device, comprising: An imaging element including an avalanche photodiode configured to photoelectrically convert an optical image; an image processing unit configured to generate an image signal based on an output signal from the image pickup element; a calculation unit configured to calculate the number of occurrences of avalanche amplification in the imaging element based on the image signal; a storage unit configured to store the cumulative value of the number of occurrences calculated by the calculation unit; as well as a control unit configured to control a parameter related to exposure when it is determined that the brightness of the image based on the image signal is outside a predetermined range, The control unit determines the predetermined range based on the accumulated value stored in the storage unit.

2. The imaging device according to claim 1, wherein: The number of occurrences is calculated by performing a predetermined transformation on the image signal.

3. The imaging device according to claim 2, wherein: The predetermined conversion includes at least one of an inverse conversion process of a digital gain, an inverse conversion process related to an inverse conversion of a linear conversion for a characteristic of the imaging element, and a subtraction process for subtracting a signal value shifted by the imaging element.

4. The imaging device according to claim 1, wherein: The number of occurrences is calculated in each of the plurality of areas.

5. The imaging device according to claim 1, wherein: The cumulative value of the number of occurrences is stored for each of the plurality of areas.

6. The imaging device according to claim 1, wherein: The control unit: calculating the number of regions or the percentage of regions where the cumulative value of the occurrence count becomes equal to or higher than a predetermined threshold, and The predetermined range is determined based on the calculated number or percentage.

7. The imaging device according to claim 6, wherein: The predetermined range is determined in such a manner that as the number or percentage is higher, the predetermined range is further expanded.

8. The imaging device according to claim 1, wherein: Whether the brightness of the image is outside the predetermined range is determined based on a difference between a reference value and a signal value indicating the brightness of the image as a signal value based on the image signal.

9. The imaging device according to claim 1, wherein: The control unit determines an exposure response coefficient based on the integrated value stored in the storage unit.

10. The imaging device according to claim 1, wherein The predetermined range is determined based on a maximum value of the number of occurrences for each of the plurality of areas.

11. The imaging device according to claim 5, wherein: The predetermined range is determined based on a cumulative value in a specific area among the cumulative values ​​that are cumulative values ​​of the number of occurrences stored for each of the plurality of areas.

12. The imaging device according to claim 1, wherein: The parameters related to exposure include at least any one of an aperture, an exposure time period, and an ND filter.

13. A control method, comprising: generating an image signal based on an output signal from an imaging element; calculating the number of occurrences of avalanche amplification in the imaging element based on the image signal; storing a cumulative value of the number of occurrences calculated in the calculation; as well as When it is determined that the brightness of the image based on the image signal is outside a predetermined range, a parameter related to exposure is controlled; Here, in the control, the predetermined range is determined based on the integrated value stored in the storage.

14. A computer-readable storage medium storing a program, wherein the program is used to cause a computer to execute a control method, the control method comprising: generating an image signal based on an output signal from an imaging element; calculating the number of occurrences of avalanche amplification in the imaging element based on the image signal; storing a cumulative value of the number of occurrences calculated in the calculation; as well as When it is determined that the brightness of the image based on the image signal is outside a predetermined range, a parameter related to exposure is controlled; Here, in the control, the predetermined range is determined based on the integrated value stored in the storage.

15. A computer program product, comprising a program, wherein the program is used to cause a computer to execute a control method, wherein the control method comprises: generating an image signal based on an output signal from an imaging element; calculating the number of occurrences of avalanche amplification in the imaging element based on the image signal; storing a cumulative value of the number of occurrences calculated in the calculation; as well as When it is determined that the brightness of the image based on the image signal is outside a predetermined range, a parameter related to exposure is controlled; Here, in the control, the predetermined range is determined based on the integrated value stored in the storage.

Images