Imaging device, control method for imaging device, and program

Abstract

To accurately grasp a change in characteristics of a SPAD sensor.SOLUTION: An imaging apparatus has a photoelectric conversion element composed of an avalanche photodiode for converting an optical image into an electrical signal. The imaging apparatus has: generation means for generating an image signal on the basis of an output signal from the photoelectric conversion element; and output means for outputting the number of times avalanche amplification occurred by performing a predetermined conversion on the basis of the image signal generated by the generation means.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an imaging device having a photoelectric conversion element formed of an avalanche photodiode. [Background technology]

[0002] One type of image sensor is the Single Photon Avalance Diode (SPAD) sensor (hereafter referred to as the SPAD sensor). The SPAD sensor uses the avalanche amplification phenomenon, in which electrons are accelerated by applying a strong electric field, and then collide with other electrons to provoke multiple electrons, causing an avalanche-like phenomenon that generates a large current. This allows weak photons that enter a pixel to be converted into a large current that can be detected as an electric charge. Because of the mechanism behind the SPAD sensor, no noise is introduced when reading out the signal, so it is expected to be used as an image sensor. In particular, because it can clearly capture subjects without being affected by noise even in dark places, it is expected to be widely used as an image sensor for surveillance applications, etc. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] C. Zhang, “SPAD requirements from consumer electronics to automotive,” Int. SPAD Sensor Workshop (2022) Summary of the Invention [Problem to be solved by the invention]

[0004] Non-Patent Document 1 reports a phenomenon in which the amount of dark current changes when a photoelectric conversion device having an APD is driven for a long time. In addition, since the dark current may change or increase depending on the number of avalanches, the degree to which the dark current changes or increases may differ depending on the region.

[0005] The present invention has been made in consideration of such problems, and has an object to accurately grasp changes in the characteristics of a SPAD sensor. [Means for solving the problem]

[0006] The invention of the present application is an imaging device having a photoelectric conversion element composed of an avalanche photodiode for photoelectrically converting an optical image, characterized in having a generation means for generating an image signal based on an output signal from the photoelectric conversion element, and an output means for outputting the number of times avalanche amplification has occurred by performing a predetermined conversion based on the image signal generated by the generation means. Effect of the Invention

[0007] According to the present invention, changes in the characteristics of a SPAD sensor can be accurately grasped. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a functional block diagram of the imaging apparatus. [Diagram 2] FIG. 2 is a schematic diagram showing an example of an equivalent circuit of a pixel that constitutes an image sensor. [Diagram 3] FIG. 4 is a schematic diagram showing an example of a process of generating an image signal. [Figure 4] 1 is a graph showing the relationship between the pulse count number, the photon number, and the image signal. [Diagram 5] 4A and 4B are diagrams showing an example of a captured image and signal values ​​of the captured image. [Figure 6] 4 is a flowchart for explaining a process executed by the imaging apparatus. [Figure 7] FIG. 1 is a diagram showing an example of the system configuration of an imaging apparatus. [Figure 8] FIG. 13 is a diagram showing an example of cumulative count values ​​by region. [Figure 9] FIG. 4 is a diagram showing an example of a notification output by the imaging device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0010] <First embodiment> The first embodiment will be described below. Fig. 2 shows a functional block diagram of an imaging device. The imaging device 1 has an avalanche photodiode for photoelectrically converting an optical image, and captures an image. The lens group 10 represents a lens group that can be controlled to achieve a desired image quality when the imaging device 1 captures an image. The lens group 10 includes a zoom lens, a focus lens, an anti-vibration lens, an aperture, an ND (Neutral Density) filter, and the like.

[0011] The image sensor section 11 is made up of photoelectric conversion elements. A light ray passing through a lens group reaches the image sensor section 11, and photoelectric conversion is performed in each pixel section on the imaging surface of the image sensor section 11 to convert the light into an electrical signal, and a digital image (so-called raw image) is generated. In this embodiment, the image sensor section 11 functions as a SPAD sensor that counts the number of photons. That is, the image sensor section 11 is made up of avalanche photodiodes for photoelectrically converting an optical image. The avalanche photodiodes form a pixel region that is arranged two-dimensionally. The process of generating a digital image from the output signal of the avalanche photodiode (hereinafter referred to as a sensor signal) will be described later.

[0012] The signal processing unit 12 performs various image processing on the digital image generated by the imaging element unit 11 to improve the image quality as an output image. The image processing includes various correction processing for improving image quality, such as removal of fixed pattern noise, brightness correction by digital gain, demosaicing processing, WB correction processing, contour enhancement processing, gamma processing, and noise reduction processing. In addition to the correction, the signal processing unit 12 also performs recognition processing for detecting a main subject area from an image for lens control such as focus and aperture control. Furthermore, the signal processing unit 12 also generates evaluation values ​​for exposure control and WB correction, and the generated evaluation values ​​are transmitted to the control calculation unit 17. The evaluation value can be, for example, an AE evaluation value. The specific processing content in the signal processing unit 12 will be described in detail later. The image corrected by the signal processing unit 12 is transmitted to the recording processing unit 13.

[0013] The recording processing unit 13 performs encoding processing on the image corrected by the signal processing unit 12, and transmits the encoded image to the recording medium 14. The recording medium 14 may be a general-purpose recording medium having a general-purpose IF and capable of being attached to / detached from a digital camera, or may be a storage device having a fixed, non-removable storage capacity that is installed inside the digital camera. The recording medium 14 writes the transmitted encoded image into a non-volatile storage area, and stores the image data.

[0014] The operation unit 15 accepts operations from the user. The accepting means may be a mechanical button, or may be an electrostatic accepting touch panel directly attached to the display unit 16. It may also be an external remote controller connected to a general-purpose terminal, or a communication device from an external terminal such as a smartphone wirelessly connected to the digital camera.

[0015] The display unit 16 converts the image sent from the control and calculation unit 17 into a format that can be displayed on a display member and displays it. In this case, the display member may be a display member directly attached to the digital camera, or may be the screen of a smartphone connected by wireless communication. It may also be a removable display member connected by a wired cable, or may be the display screen of a terminal connected to a network via a LAN cable.

[0016] The control calculation unit 17 is composed of a known CPU or the like, receives operation signals from the digital camera through the operation unit 15, generates control information for each block, and transmits the information to the lens group 10, the image sensor unit 11, the signal processing unit 12, the recording processing unit 13, and the recording medium 14. Furthermore, the control calculation unit 17 also executes image generation control to be sent to the display unit 16. Moreover, the memory 18 is an area for storing data necessary for the control calculation in the control calculation unit 17, and stores data during the calculation and the calculation results in the memory. The control calculation unit 17 executes the calculation process while appropriately referring to the data in the memory 18. Moreover, the control calculation unit 17 is connected to the communication unit 19, and can also output the calculation results to an external device connected through the communication unit 19.

[0017] Next, the pixel of the SPAD sensor in the image sensor unit 11 will be described with reference to the equivalent circuit diagram of the pixel in FIG. 2. Each pixel is composed of a photodiode 20, a quench resistor 21, and a buffer 22, and the photodiode 20 is a SPAD. When a reverse voltage exceeding the breakdown voltage is applied to the photodiode 20, the photodiode 20 operates in a Geiger mode triggered by the input of a photon. Therefore, a large reverse bias voltage equal to or greater than the breakdown voltage of the voltage VH applied to the SPAD is required. When a photon is input to the photodiode 20, an avalanche current is generated. The quench voltage VQ applied to the quench resistor 21 is configured to be variable, and the reverse bias voltage is lowered by the quench resistor value with respect to the avalanche current generated in the photodiode 20, making it possible to control the end of the avalanche amplification. Here, the specific configuration of the quench resistor 21 may be a MOSFET or another semiconductor element having a quench function. The buffer 22 has a threshold voltage set, and outputs an H level for a certain period of time according to the fluctuation of the input voltage, so that it has a waveform shaping function. The signal shaped by the buffer 22 in this way is output as a pulse waveform. The number of pulse waveforms of this pulse signal is counted by a counter circuit prepared in the subsequent stage, and the number of pulses within a certain period of time is output as a sensor signal value of avalanche amplification. This sensor signal value is the number of photon counts per unit time, and can be considered as the number of avalanche amplifications. Here, the circuit configuration of each pixel of the SPAD sensor has been described based on a simple configuration having a photodiode 20, a quench resistor 21, and a buffer 22, but the pixel circuit configuration may also include other circuit elements or semiconductors.

[0018] Next, a flowchart of the signal processing unit 12 for identifying an image signal value from a sensor signal value will be described with reference to FIG. 3. Some of the functional blocks shown in FIG. 3 are realized by the control calculation unit 17, which is a CPU included in the photoelectric conversion device 100, executing a computer program (control method) stored in a memory such as the storage medium 14. However, some or all of them may be realized by hardware. As the hardware, a dedicated circuit (ASIC), an FPGA (field programmable gate array), a processor (reconfigurable processor, DSP), etc. may be used. Moreover, each functional block shown in FIG. 3 does not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The same applies to FIG. 6 described later.

[0019] First, in S300, the signal processing unit 12 obtains the number of pulses counted for each pixel in the imaging element unit 11 as an output signal for each pixel.

[0020] Next, the signal processing unit 12 obtains a first image signal value by performing linear conversion on the acquired sensor signal value (number of pulses) of each pixel using a predetermined function according to the characteristics of the image sensor unit 11. In the SPAD sensor, there is a limit to the time resolution of photon counting, and when the number of photons in a unit time is extremely large, photons may not be separated one by one and may be counted. As a result, when the number of photons is large, the number of pulses and the number of photons do not change linearly, and have a nonlinear correlation as shown in FIG. 4(a). Here, the nonlinear characteristic is expressed as a probability function of the number of pulses. There is a nonlinear correlation between the number of photons and the number of pulses, and a linear number of photons can be calculated by converting the observed number of pulses using the inverse function of the nonlinear characteristic. Inverse conversion characteristics that can convert the number of pulses to the number of input photons are shown in FIG. 4(b). In this way, an image signal value that is linear to the number of photons can be obtained by converting the sensor signal value with a predetermined probability function using the inverse characteristic of the probability function. Note that the count of the sensor signal value becomes saturated when a certain amount of light or more is received, so a predetermined linear conversion may be performed only when the sensor signal value is equal to or greater than a predetermined value.

[0021] Next, in S302, the signal processing unit 12 adjusts the brightness of the image by amplifying the first image signal value by digital gain as one means of exposure control. This makes it possible to obtain a second image signal value that changes linearly with respect to the number of photons and is maintained at an appropriate brightness. In FIG. 3, linear conversion and digital gain are mentioned as conversion processing to be performed on the number of pulses for each pixel, but other processing may be performed. For example, FPN correction for detecting and correcting defective pixels, shading correction for correcting unevenness in the black level within a screen, optical correction for correcting deterioration due to the characteristics of an optical lens, etc. may be included. Furthermore, white balance processing for adjusting the WB (white balance) of the image, edge enhancement processing, and noise reduction processing may be included.

[0022] A feature of SPAD sensors is that they can count the number of photons by generating a large current due to the avalanche amplification phenomenon in response to the input photons. On the other hand, in order to generate the avalanche amplification phenomenon, a reverse bias voltage exceeding the breakdown voltage must be applied, and a large current flows when a large voltage is applied. In the case of using the sensor as an image sensor, the sensor will capture 30 or more frames per second in video recording, and a large current will repeatedly flow through the circuit elements of each pixel, placing a very large load on the sensor. In addition, the large avalanche current may cause a localized temperature rise, and dark electrons generated in the trap level may flow into the avalanche multiplication region, so the amount of dark current may change in each region depending on the number of times that a large current flows due to avalanche amplification. Therefore, if the number of avalanche amplifications in each pixel or region of the SPAD sensor is accumulated and counted, it becomes possible to accurately grasp the change in image quality due to the change in dark current. Counting the number of avalanche amplifications for each pixel has the advantage of being able to capture changes in image characteristics with higher accuracy. On the other hand, when the number of pixels in an image sensor is large, recording the number of avalanche amplifications on a pixel-by-pixel basis results in a large amount of data. Therefore, by counting the number of avalanche amplifications by region, it is possible to reduce the amount of data while still adequately understanding changes in image characteristics.

[0023] First, consider the case where the second image signal values ​​are acquired and accumulated for each region of an image. FIG. 5(a) shows an example of an image captured by a SPAD sensor. In the image, a forest, a backlit building, the sky, and the sun are captured. The sun has a very high second image signal value, followed by the sky, the forest, and the backlit building. FIG. 5(b) shows the second image signal values ​​acquired for each region by dividing the image into 4 horizontal and 4 vertical regions. Here, the second image signal value acquired from the image region where the sun is reflected is very large, and the image signal value acquired from the image region where the backlit building is reflected is small. Here, the second image signal value may be acquired for each color of the Bayer image, or may be acquired for each color and brightness of a YCC or YUV image. In addition, the division number of the image signal values ​​for each region does not need to be 4×4, and may be acquired in 8×8 or 16×16 regions if the memory area allows. Also, although an example of acquiring the average value of the second image signal value by region has been shown here, it is also possible to acquire the image signal value within the region as a statistical value such as the sum or median, etc. Note that, although an example of calculating the cumulative value by region will be described, if there is room in the recording area and computational costs, it is also possible to acquire and compute the signal value by pixel, rather than by region.

[0024] As mentioned above, the second image signal values ​​obtained for each region are converted from the number of pulses. Therefore, the image signal values ​​are not strictly the same as the number of avalanche amplifications. In the flow shown in Figure 3, digital gain and linear conversion were performed, but to calculate the number of avalanche amplification pulses from the image signal values, it is necessary to perform an inverse conversion process to return the converted values ​​to their original state and then estimate the number of avalanche amplifications. The flow of the process to output an estimated value of the number of avalanche amplifications from the image signal is explained below with reference to the flowchart in Figure 6.

[0025] First, the camera is started. In S600, the control and calculation unit 17 acquires a second image signal value by capturing an image using the SPAD sensor. The captured image is based on the image signal value that has been subjected to the predetermined conversion processing of linear conversion and digital gain in FIG. 3. In S601, the control and calculation unit 17 performs an inverse conversion of the amplification by the digital gain on the acquired image signal value. Note that, if processing other than the digital gain has been performed, a process is performed to reverse the conversion processing. In S602, the control and calculation unit 17 acquires a sensor estimate value that is nonlinear with respect to the number of photons by performing an inverse conversion of the linear conversion on the image signal value converted in S601. The sensor estimate value is a value that estimates the number of avalanche amplifications for each region. Also, the number of avalanche amplifications that have occurred since the start of imaging is called an accumulated value. In S666, it is determined whether or not imaging by the imaging device has ended. If imaging is to be continued, the sensor estimate value counted this time is added to the accumulated value so far and recorded, and S600 to S602 are performed again. If an end instruction from the user is received, imaging is ended. The series of conversion processes (digital gain) may be performed for each predetermined number of frames, or may be performed for each frame to obtain the number of avalanche amplifications for each frame. Here, the number of pulses converted in the repeated process is accumulated by region as sensor estimates while the camera is activated and continues to capture images. By continuing this, it is possible to grasp the extent to which each pixel of the SPAD sensor repeats avalanche amplification for each region. Here, since the accumulated value can become an astronomical figure depending on the brightness of the subject of the camera and the operation period, the accumulated value may be stored in a logarithmic representation or a floating-point representation. It may also be divided into an exponent part and an integer part and stored in a non-volatile memory. In addition, with regard to the accumulation of sensor estimates, in applications such as surveillance cameras that continue to capture a fixed subject for a long period of time, it is not necessary to accumulate the sensor estimates for all consecutive frames, and it is also possible to calculate and accumulate the number of pulses for each region in the image at regular intervals.

[0026] The use of the estimated number of occurrences of avalanche amplification (accumulated value) will be mentioned. Here, an example of an imaging system including an imaging device 1, which is a surveillance camera, will be described with reference to FIG. 7. FIG. 8 shows an example of the accumulated value of avalanche amplification by region. The provider of the SPAD sensor can predetermine the accumulated value of avalanche amplification as a guideline for the timing of sensor replacement as a predetermined number. For example, when the maximum value of the accumulated values ​​by region is focused on, if the maximum value 350M exceeds a predetermined number M [times], it can be determined that the deterioration has progressed to the extent that the sensor replacement guideline is exceeded in the region within the image. That is, the control calculation unit 17 may determine whether the accumulated value of avalanche amplification has exceeded the predetermined number, and output information indicating the state of the imaging element unit 11 according to the determination result. Specifically, when the maximum value of the accumulated value by region exceeds the predetermined number, the imaging device 1 may display a warning to encourage the user to replace the sensor, as shown in FIG. 9(a). The information indicating the state of the imaging element unit 11 may be information indicating whether the sensor of the imaging device 1 needs to be replaced.

[0027] In addition, if the expected service life of the imaging device is determined in advance, the content of the warning may be changed by comparing the current total operating time with the cumulative value by area. For example, a warning may be issued based on whether the cumulative value is expected to exceed M times within the service life if the imaging device is used at the current pace.

[0028] In addition, as a use of the accumulated value, the accumulated value may be directly displayed on a display device in an operation mode used by the user support department. As shown in FIG. 9(b), the accumulated value of the avalanche amplification may be displayed alongside the endurance count of durable parts such as the aperture and ND, and the accumulated value may be used as a guide for sensor replacement in the user support department based on the accumulated value. Here, the total number of accumulated values ​​by area may be displayed, or only the maximum accumulated value may be displayed. Furthermore, it may be used as a display device other than the imaging device. That is, the imaging device and a communication terminal may be connected, and the accumulated value may be displayed on the display device of the communication terminal, or the accumulated value may be stored in the terminal through data communication. For example, as shown in the system configuration example of FIG. 7, the imaging device 1 may be connected to a wireless or wired network 2, and the video may be recorded and analyzed in the server 3, and monitored by the external device 4. In that case, the accumulated value will be output to the external device 4 through the communication unit 11 of the imaging device. In addition, the output value may be the accumulated value itself, an approximate value obtained by rounding the accumulated value, or something like an index of the degree of deterioration based on the accumulated value. That is, the imaging device 1 has a setting means for setting an operation mode for notifying a user of the number of avalanche amplifications in the photoelectric conversion element estimated in a plurality of images. Furthermore, an information processing device such as a communication terminal connected to the imaging device via a network or the like has an output means for outputting notification information based on the number of avalanche amplifications when the imaging device is set to the operation mode.

[0029] <Other embodiments> Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0030] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a recording medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions. [Explanation of symbols]

[0031] 1. Imaging device 10 Lens Group 11. Image sensor section 12 Signal Processing Section 13 Recording processing section 14 Recording media 15 Control section 16 Display 17 Control and calculation section 18 Memory 19 Communications Department

Claims

1. An imaging device having a photoelectric conversion element configured with an avalanche photodiode for photoelectrically converting an optical image, a generating means for generating an image signal based on an output signal from the photoelectric conversion element; and an output unit that outputs the number of occurrences of avalanche amplification by performing a predetermined conversion based on the image signal generated by the generation unit.

2. the generating means generates the image signal for each region of the photoelectric conversion element; 2. The imaging apparatus according to claim 1, wherein the output means outputs the number of occurrences of avalanche amplification for each of the regions.

3. the generating means generates the image signal for each pixel in the photoelectric conversion element; 2. The imaging apparatus according to claim 1, wherein the output means outputs the number of times avalanche amplification has occurred for each pixel.

4. 2. The imaging apparatus according to claim 1, wherein the output unit outputs information to inform a user based on the number of occurrences of avalanche amplification.

5. Further comprising a recording means for recording the operation time of the photoelectric conversion element, 4. The imaging device according to claim 3, wherein the output means determines information to be notified to a user based on a cumulative value of the number of times avalanche amplification occurs in the plurality of image signals and an operating time of the photoelectric conversion element.

6. the output signal indicates the number of pulses of avalanche amplification in the photoelectric conversion element, The image signal is a signal obtained by converting the output signal through a predetermined conversion process, 2. The imaging apparatus according to claim 1, wherein the predetermined conversion is an inverse conversion of the predetermined conversion process performed on the image signal.

7. 7. The imaging device according to claim 6, wherein the predetermined conversion process includes at least one of a digital gain and a linear conversion of a characteristic value of the photoelectric conversion element.

8. 8. The imaging apparatus according to claim 7, wherein the predetermined conversion process is performed based on the digital gain determined for each predetermined number of frames.

9. The predetermined conversion processing includes FPN correction, shading correction, and optical correction.

8. The imaging apparatus according to claim 7, further comprising at least one of a white balance process, an edge enhancement process, and a noise reduction process.

10. 2. The imaging apparatus according to claim 1, wherein the output means outputs a cumulative value of the number of occurrences of the avalanche amplification based on a plurality of the image signals.

11. 2. The imaging apparatus according to claim 1, further comprising a recording means for recording a cumulative value of the number of occurrences of avalanche amplification output by said output means.

12. 12. The imaging apparatus according to claim 11, wherein the output means outputs information to inform a user when a cumulative value of the number of occurrences of avalanche amplification exceeds a predetermined number.

13. 5. The imaging apparatus according to claim 4, wherein the output unit outputs information to be notified to a user based on a maximum value among the accumulated values ​​of the number of occurrences of avalanche amplification for each region.

14. a transmitting means for transmitting data relating to the photoelectric conversion element, the data being determined based on the cumulative value of the number of occurrences of avalanche amplification output by the output means, to an external device; 2. The imaging apparatus according to claim 1, wherein the data relating to the photoelectric conversion element is information indicating a state of the photoelectric conversion element or whether or not replacement is required.

15. An imaging device having a photoelectric conversion element configured with an avalanche photodiode for photoelectrically converting an optical image, A generating means for generating an image based on an output signal from the photoelectric conversion element; and an output unit that outputs information to inform a user based on the number of avalanche amplifications in the photoelectric conversion element estimated in the plurality of images.

16. An information processing system including an imaging device having a photoelectric conversion element configured with an avalanche photodiode for photoelectrically converting an optical image, and an output device that outputs notification information for a user, The imaging device includes: A generating means for generating an image based on an output signal from the photoelectric conversion element; a setting unit for setting a mode for notifying a user of the number of avalanche amplifications in the photoelectric conversion element estimated in the plurality of images; The output device includes: an output unit that outputs the notification information based on the number of avalanche amplifications when the imaging device is set to the mode;

17. A method for controlling an imaging device having a photoelectric conversion element constituted by an avalanche photodiode for photoelectrically converting an optical image, comprising: a generating step of generating an image signal based on an output signal from the photoelectric conversion element; and an output step of outputting the number of occurrences of avalanche amplification by performing a predetermined conversion based on the generated image signal.

18. A computer program for controlling each unit of the photoelectric conversion device according to any one of claims 1 to 15 by a computer.

Images