Photovoltaic conversion device, control method, and computer program

Abstract

To provide a photoelectric conversion device, a control method for the same, and a computer program, capable of suppressing deterioration of an image quality as well as having an avalanche photodiode.SOLUTION: A photoelectric conversion device 300 includes: a photoelectric conversion element that is constructed by an avalanche photodiode for performing a photoelectric conversion of an optical image; an image generation part that generates a first image on the basis of a signal acquired by the photoelectric conversion element; a storage part that stores first characteristic information of the photoelectric conversion element; a first correction processing part that performs a first correction processing for correcting a liner property of the first image by using the first characteristic information; and a second correction processing part that performs a second correction processing for performing a pixel interpolation on the basis of both of the first characteristic information and information on the first image.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a photoelectric conversion device having an avalanche photodiode, a control method, and a computer program. [Background technology]

[0002] 2. Description of the Related Art In recent years, a photoelectric conversion device has been proposed that digitally counts the number of photons arriving at an avalanche photodiode (APD) and outputs the counted value from a pixel as a photoelectrically converted digital signal.

[0003] It is known that a phenomenon called avalanche emission occurs in photoelectric conversion devices having APDs (for example, Non-Patent Document 1). When avalanche emission occurs, the generated secondary electrons enter adjacent pixels, increasing the count number of the adjacent pixel values ​​and causing erroneous counting. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] I. Rech et al., “Optical crosstalk in single photon avalanche diode arrays: a new complete model”, OpEx 16(12), 2008 Summary of the Invention [Problem to be solved by the invention]

[0005] SUMMARY OF THE PRESENT EMBODIMENTS Accordingly, an object of the present invention is to provide a photoelectric conversion device that has an avalanche photodiode and is capable of suppressing degradation in image quality. [Means for solving the problem]

[0006] A photoelectric conversion device according to one aspect of the present invention comprises: a photoelectric conversion element configured with an avalanche photodiode for photoelectrically converting an optical image; an image generating means for generating a first image based on a signal acquired by the photoelectric conversion element; an acquisition means for acquiring first characteristic information related to crosstalk between pixels of the photoelectric conversion element; a first correction processing means for performing a first correction process for correcting linearity of the first image by using the first characteristic information; and second correction processing means for performing a second correction process for pixel interpolation based on both the first characteristic information and the information of the first image. Effect of the Invention

[0007] According to the present invention, it is possible to provide a photoelectric conversion device that has an avalanche photodiode and is capable of suppressing deterioration in image quality. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing a configuration example of a photoelectric conversion element according to a first embodiment. [Diagram 2] 2 is a diagram showing a configuration example of a sensor substrate 11. FIG. [Diagram 3] 2 is a diagram showing an example of the configuration of a circuit board 21. FIG. [Figure 4] 2 and 3, and shows an equivalent circuit of a pixel 101 and a signal processing circuit 103 corresponding to the pixel 101. [Diagram 5] 2 is a diagram illustrating a relationship between the operation of an APD 201 and an output signal. FIG. [Figure 6] 1 is a functional block diagram of a photoelectric conversion device 300 according to a first embodiment. [Figure 7] 1 is a diagram showing a probability distribution of occurrence of erroneous counting between adjacent pixels of the photoelectric conversion element 100 as first array data. [Figure 8] 4 is a flowchart relating to signal processing in the photoelectric conversion device according to the first embodiment. [Figure 9]5A to 5C are diagrams illustrating a first correction process according to the first embodiment. [Figure 10] 6A to 6C are diagrams illustrating a second correction process according to the first embodiment. [Figure 11] 10 is a flowchart of signal processing in a photoelectric conversion device according to the second embodiment. [Figure 12] 10A to 10C are diagrams illustrating a second correction process according to the second embodiment. [Figure 13] 11 is an equivalent circuit diagram corresponding to a pixel of a photoelectric conversion element according to embodiment 3. FIG. [Figure 14] 10 is a drive timing chart of a photoelectric conversion element according to the third embodiment. [Figure 15] 10 is a signal processing flowchart of a photoelectric conversion device according to a third embodiment. [Figure 16] 13A to 13C are diagrams illustrating a first correction process according to the third embodiment. [Figure 17] 10 is a flowchart of signal processing in a photoelectric conversion device according to a fourth embodiment. [Figure 18] FIG. 11 is a functional block diagram of a photoelectric conversion system according to a fifth embodiment using the photoelectric conversion devices of the first to fourth embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0010] <Embodiment 1> 1 is a diagram showing an example of the configuration of a photoelectric conversion element according to the first embodiment. In the following, a photoelectric conversion device having a so-called laminated structure in which a photoelectric conversion element 100 is configured by laminating and electrically connecting two substrates, a sensor substrate 11 and a circuit substrate 21, will be described as an example. However, a so-called non-laminated structure in which the configuration included in the sensor substrate and the configuration included in the circuit substrate are arranged on a common semiconductor layer may also be used. The sensor substrate 11 includes a pixel region 12. The circuit substrate 21 includes a circuit region 22 that processes a signal detected in the pixel region 12.

[0011] 2 is a diagram showing a configuration example of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and columns. Each pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD). The number of rows and columns of the pixel array constituting the pixel region 12 is not particularly limited.

[0012] Fig. 3 is a diagram showing an example of the configuration of the circuit board 21. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by each photoelectric conversion unit 102 in Fig. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114.

[0013] The vertical scanning circuit 110 receives a control pulse supplied from a control pulse generating unit 115, and sequentially supplies the control pulse to a plurality of pixels arranged in the row direction. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

[0014] The signal output from the photoelectric conversion unit 102 of each pixel is processed by each signal processing circuit 103. The signal processing circuit 103 is provided with a counter, a memory, etc., and a digital value is stored in the memory. The horizontal scanning circuit 111 inputs a control pulse that sequentially selects each column to the signal processing circuit 103 in order to read the signal from the memory of each pixel in which the digital signal is stored.

[0015] A signal is output to the vertical signal line 113 from the signal processing circuit 103 of the pixel of the row selected by the vertical scanning circuit 110. The signal output to the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the read circuit 112 and the output circuit 114.

[0016] 2 and 3, a plurality of signal processing circuits 103 are arranged in a region overlapping the pixel region 12 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so as to overlap between an end of the sensor substrate 11 and an end of the pixel region 12 in a plan view.

[0017] In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region arranged around the pixel region 12. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

[0018] The arrangement of the vertical signal lines 113, the readout circuits 112, and the output circuits 114 are not limited to the example shown in Fig. 3. For example, the vertical signal lines 113 may be arranged extending in the row direction, and the readout circuits 112 may be arranged at the ends of the vertical signal lines 113. Also, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit, and one signal processing unit may be shared by a plurality of photoelectric conversion units to perform signal processing sequentially.

[0019] FIG. 4 is a diagram showing an equivalent circuit of the pixel 101 in FIG. 2 and FIG.

[0020] The APD 201 included in the photoelectric conversion unit 102 generates a pair of electric charges according to incident light by photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VL (first voltage) is supplied. The other of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VH (second voltage) higher than the voltage VL is supplied.

[0021] In Fig. 4, one node of the APD 201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD 201 so that the APD 201 performs avalanche multiplication. By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

[0022] When a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the voltage difference between the anode and cathode is close to or less than the breakdown voltage. An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V.

[0023] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a selection circuit 212. The quench element 202 is connected to a power supply line to which a driving voltage VH is supplied and one of the anode and cathode nodes of the APD 201.

[0024] The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, and has a function of suppressing avalanche multiplication by suppressing the voltage supplied to the APD 201 (quench operation).The quench element 202 also has a function of returning the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0025] 4 shows an example in which the signal processing circuit 103 has a waveform shaping unit 210, a counter circuit 211, and a selection circuit 212 in addition to the quench element 202. However, it is sufficient for the signal processing circuit 103 to have at least one of the waveform shaping unit 210, the counter circuit 211, and the selection circuit 212 in addition to the quench element 202.

[0026] The waveform shaping unit 210 shapes the voltage change of the cathode of the APD 201 obtained when a photon is detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. Although an example in which one inverter is used as the waveform shaping unit 210 is shown in Fig. 4, a circuit in which a plurality of inverters are connected in series may be used, or another circuit having a waveform shaping effect may be used.

[0027] The counter circuit 211 counts the pulse signal output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a drive line 213, the signal held in the counter circuit 211 is reset.

[0028] 3 through a drive line 214 (not shown in FIG. 3) in FIG. 4, and switches between electrical connection and non-connection between the counter circuit 211 and the vertical signal line 113. The selection circuit 212 includes, for example, a buffer circuit for outputting a signal, and outputs an output signal from the counter circuit 211 of the pixel to the vertical signal line 113.

[0029] Note that electrical connections may be switched by disposing a switch such as a transistor between the quench element 202 and the APD 201 or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the voltage VH or the voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0030] 5 is a diagram showing a schematic diagram of the relationship between the operation of the APD 201 and the output signal. The input side of the waveform shaping unit 210 is nodeA, and the output side is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops.

[0031] When the voltage drop becomes even larger and the potential difference applied to APD 201 becomes smaller, avalanche multiplication of APD 201 stops as at time t2, and the voltage level at nodeA does not drop below a certain value. After that, between time t2 and time t3, a current that compensates for the voltage drop from voltage VL flows to nodeA, and at time t3, nodeA is stabilized to its original potential level. At this time, the portion of the output waveform at nodeA that exceeds a certain threshold is shaped by waveform shaping section 210 and is output as a pulse signal at nodeB.

[0032] The photoelectric conversion device 300 according to the first embodiment will be described below. Fig. 6 is a functional block diagram of the photoelectric conversion device 300 according to the first embodiment. Some of the functional blocks shown in Fig. 6 are realized by causing a computer (not shown) included in the photoelectric conversion device 300 to execute a computer program stored in a memory serving as a storage medium (not shown).

[0033] However, some or all of them may be realized by hardware. A dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) may be used as the hardware. Furthermore, the functional blocks shown in Fig. 6 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation regarding Fig. 6 also applies to Fig. 18.

[0034] 1 to 5, an imaging optical system 301, and a signal processing unit 302 that processes an image signal acquired by the photoelectric conversion element 100. The photoelectric conversion element 100 is configured with an avalanche photodiode for photoelectrically converting an optical image. In the first embodiment, a case will be described in which each pixel of the photoelectric conversion element 100 does not have a color filter, and is a so-called monochrome sensor.

[0035] The signal processing unit 302 has an image generating unit 303 that generates a first image from an image signal acquired by the photoelectric conversion unit, a first correction processing unit 304, and a second correction processing unit 305. The signal processing unit 302 also has a memory unit 306 as a storage means that stores first array data related to the first characteristic information of the photoelectric conversion element 100.

[0036] The above-mentioned first characteristic information is information on crosstalk characteristics between pixels caused by the avalanche light emission phenomenon of the photoelectric conversion element 100. The storage unit 306 may download the first characteristic information (first array data, etc.) from an external server or the like and temporarily store it.

[0037] Then, in the first correction processing unit, the linearity of the first image is corrected by performing miscounting correction (first correction processing) using first characteristic information (first array data, etc.) of the photoelectric conversion element 100.

[0038] Moreover, the second correction processing unit performs a second correction process for pixel interpolation (defect correction process) of a signal of a defective pixel using information of the first image and information of the first characteristic information (first array data, etc.) of the photoelectric conversion element 100. Note that the first characteristic information regarding the crosstalk characteristic between pixels caused by the avalanche light emission phenomenon of the photoelectric conversion element may be array information or a function.

[0039] 7 is a diagram showing, as first array data, a probability distribution of occurrence of miscounting between adjacent pixels of the photoelectric conversion element 100, and shows an example of the first array data corresponding to the first characteristic information of the photoelectric conversion element 100. The first array data (first characteristic information) as shown in Fig. 7 is stored in the storage unit 306, and this first array data is used to perform correction in the first correction processing unit 304 as the first correction processing means and the second correction processing unit 305 as the second correction processing means.

[0040] The first array data or the like serving as the first characteristic information may be stored as a table in the storage unit 306 or the like, or may be saved as a function.

[0041] As described in Non-Patent Document 1, when a pixel is an avalanche photodiode, avalanche light emission phenomenon causes erroneous counting in adjacent pixels, that is, crosstalk between adjacent pixels (hereinafter referred to as light emission crosstalk).

[0042] The influence of a photon incident on one pixel on an adjacent pixel is determined by the probability of occurrence of light emission crosstalk. The probability of occurrence of light emission crosstalk is determined by the pixel structure of the photoelectric conversion element, and therefore the probability of occurrence of light emission crosstalk can be predicted by the pixel structure of the photoelectric conversion element. Therefore, in the photoelectric conversion device 300 according to the first embodiment, the information on the probability of occurrence of light emission crosstalk is used to perform signal processing that suppresses the influence of erroneous counting, thereby improving image quality.

[0043] Fig. 8 is a flowchart relating to signal processing of the photoelectric conversion device according to embodiment 1. Note that the computer in the photoelectric conversion device 300 executes a computer program stored in the memory to perform the operations of each step of the control method for the photoelectric conversion device shown in the flowchart of Fig. 8.

[0044] First, in step S401 (image generation step), a first image is generated by the image generation unit 303 as an image generating means based on a signal acquired by the photoelectric conversion element 100. After that, the process branches into a first correction process and a second correction process.

[0045] First, the first correction process will be described. In step S402, a convolution operation is performed on the first image generated by the image generation unit 303 to convolute the second array data generated by the first correction processing unit 304 based on the first array data, thereby generating a second image.

[0046] As described above, since the probability of occurrence of light emission crosstalk is predictable, by performing a convolution operation, the second image becomes a signal representing a miscount caused by light emission crosstalk. The first characteristic information (first array data, etc.) may be acquired from the storage unit 306 or an external server, etc. The second array data may also be stored as a table or a function in the storage unit 306, etc. Here, step S402 functions as an acquisition step (acquisition means) for acquiring the first characteristic information of the photoelectric conversion element.

[0047] Next, in step S403, the first correction processing unit 304 serving as a first correction processing means subtracts the second image from the first image to generate a third image. As described above, since the second image is a signal representing a miscount caused by light emission crosstalk, the third image is an image obtained by restoring a signal that would be obtained if no miscount caused by light emission crosstalk occurred.

[0048] Here, steps S402 and S403 function as a first correction processing step (first correction processing means) that performs a first correction process to correct the linearity of the first image using the first array data.

[0049] 9 is a diagram for explaining the first correction process according to the first embodiment, where (A) shows the first image, (B) shows the second array data, (C) shows the second image, and (D) shows the third image. The second array data used in step S402 is the first array data in FIG. 7 with the pixel value set to 0.

[0050] However, as long as it is based on the first array data, it may be different from (B). In addition, in step S402, instead of performing a convolution operation, the first image and the second array data may be Fourier transformed and then the product may be obtained. In other words, the first correction process may be a process in which the result of a predetermined operation (convolution operation, Fourier operation and multiplication, etc.) using the first characteristic information of the photoelectric conversion element and the first image is subtracted from the first image.

[0051] Next, the second correction process will be described. The second correction processing unit performs correction for defective pixels. As described above, when the pixel is an avalanche photodiode, the effect of the defective pixel propagates to the surrounding pixels due to light emission crosstalk. Such a defect is called a cluster defect. The second correction processing unit improves image quality by performing signal processing that suppresses the effect of miscounting due to cluster defects using information on the occurrence probability of light emission crosstalk.

[0052] First, in step S404, pixels having a higher output level than surrounding pixels are extracted as defective pixels from the first image generated by the image generating unit 303. Fig. 10 is a diagram for explaining the second correction process according to the first embodiment.

[0053] 10, (A) shows the first image, and (B) shows the result of extraction of defective pixels extracted in step S404. The positions and levels of defective pixels can be determined by capturing a dark image in advance, storing the positions and output levels of pixels with outputs equal to or greater than a predetermined threshold as address data in a memory such as the storage unit 306, and using these.

[0054] In step S405, crosstalk estimation is performed. That is, for each defective pixel, third array data is generated based on the output value of the defective pixel and the first array data. The third array data is array data that estimates the extent to which pixels surrounding the extracted defective pixel are affected by crosstalk from the defective pixel.

[0055] Specifically, the third array data that estimates the effect of crosstalk can be generated by multiplying the level of the defective pixel extracted from the first image (the output value of the first image) by the first array data. Note that the third array data is not limited to this, as long as it is array data based on the first image and the first array data. (C) of Figure 10 shows the first array data, and (D) shows the third array data.

[0056] Next, in step S406, a fourth image is generated by performing defect correction on pixels where the value (crosstalk estimate) of the third array data generated for each defective pixel is estimated to be equal to or greater than a predetermined first threshold value.

[0057] In this way, steps S404 to S406 function as a second correction processing step (second correction processing means) that performs a second correction process for pixel interpolation based on both the first array data and the information of the first image. In (E) of Fig. 10, pixels (pixels to be subjected to defect correction) whose crosstalk estimated amount is equal to or greater than the first threshold value are indicated by a cross.

[0058] Specifically, the output values ​​of pixels whose values ​​of the third array data are equal to or greater than the first threshold are interpolated for the third image generated in step S403 using output values ​​of multiple surrounding pixels whose values ​​of the third array data are less than the first threshold. That is, the output values ​​of the third image are corrected based on the third array data.

[0059] The interpolation process may be carried out by applying a median filter to the pixels surrounding the pixel to be interpolated, excluding pixels whose third array data is equal to or greater than the first threshold value.

[0060] The kernel size of the median filter should be 3x3 or more. However, because cluster defects can cause the effect of a defective pixel to extend to surrounding pixels, it is preferable to use a kernel size of 5x5 or more.

[0061] Also, the size of the median filter may be changed according to the ratio of pixels around the pixel to be interpolated that have a crosstalk estimation amount equal to or greater than the first threshold. That is, the more pixels around the pixel to be interpolated that have the third array data equal to or greater than the first threshold, the larger the kernel size is, thereby increasing the number of pixels that can be used for correction (pixels whose third array data are less than the first threshold), thereby improving the accuracy of the interpolation. Note that instead of the median filter, a bilateral filter, a Gaussian filter, or the like may be used.

[0062] As shown in Fig. 7, the first array data as the first characteristic information may be one-dimensional or two-dimensional as long as it includes two or more pieces of data, but it is preferable that it is two-dimensional array data. Furthermore, considering the symmetry of the crosstalk probability, it is preferable that the matrix has an odd number of rows and columns and is symmetrical up, down, left, and right about the center. Since the crosstalk probability value increases closer to the pixel, the crosstalk matrix has a distribution that has a peak value in the center and changes monotonically toward the periphery.

[0063] Specifically, when one row or one column of a crosstalk matrix, which is two-dimensional array data, is extracted as one-dimensional data, the one-dimensional data has a distribution that monotonically decreases from a central peak value toward both data ends. That is, the one-dimensional data corresponding to at least one row or column of the two-dimensional array data has a peak value in the center. Furthermore, other one-dimensional data that share the peak value of the extracted one-dimensional data and are arranged in a direction intersecting the one-dimensional data also have a distribution that monotonically decreases from the peak value toward the data ends.

[0064] In addition, in FIG. 8, an example is shown in which the address data stored in advance in the memory is used to extract the flaws in step S404, but the flawed pixels and their output levels may be extracted from the first image. Specifically, a median filter is applied to the first image to generate a fifth image. Then, pixels whose output difference between the first image and the fifth image is equal to or greater than the second threshold value are extracted as flawed pixels, and the difference between the first image and the fifth image is set as the output level of the flawed pixels. That is, the flawed pixels may be extracted by comparing the output of each pixel of the first image with the output of the surrounding pixels.

[0065] Extracting defective pixels and their output levels from the first image is preferable since it does not require a memory for storing defective addresses as compared to using address data, and therefore the configuration of the photoelectric conversion device 300 is simplified.

[0066] On the other hand, when extracting defective pixels from the first image using a median filter, there is a possibility that defective pixels will be erroneously detected if a subject such as a bright spot is photographed. Therefore, in order to extract defective pixels and their output levels with high accuracy, it is preferable to use address data. Note that, although the above shows the case where a median filter is used, defective pixels may also be extracted by comparing the output value of the pixel with the average output value of the surrounding pixels.

[0067] In this way, when the first image is used to extract pixels to be corrected in the second correction process, steps S404 and S405 may be integrated. That is, for each pixel of the first image, a sixth image is generated by subtracting the fifth image, which has been subjected to a median filter, from the first image.

[0068] Then, the third array data is obtained by multiplying each pixel of the sixth pixel by the first array data, and a process of pixel-interpolating at least one pixel of the third array data equal to or greater than the first threshold value with a plurality of pixels of the third array data less than the first threshold value is performed. The reason is explained below.

[0069] As mentioned above, the closer the pixel is to the crosstalk probability, the larger the value becomes. Also, the third array data is array data that indicates the degree to which the pixels surrounding the extracted defective pixel are affected by crosstalk from the defective pixel. Therefore, in the third array data, the value of the pixel is larger than the values ​​of the surrounding pixels.

[0070] Therefore, if a pixel in the third array data that is equal to or greater than the first threshold value is to be subjected to interpolation, the pixel is also to be subjected to interpolation. In this way, when the first image is used to extract pixels to be corrected in the second correction process, it is preferable to integrate steps S404 and S405, since this eliminates the need for the step of extracting defects in step S404 and reduces the circuit scale of the correction process.

[0071] In this way, in the photoelectric conversion device of this embodiment, the first correction process suppresses miscounting at adjacent pixels caused by the avalanche light emission phenomenon, and the second correction process suppresses miscounting at adjacent pixels also caused by the avalanche light emission phenomenon, particularly miscounting caused by cluster scratches, thereby improving image quality.

[0072] On the other hand, even if the second correction process is not performed and only the first correction process is performed, it is possible to suppress miscounting caused by cluster scratches. However, it is preferable to perform the second correction process, because it is possible to further suppress miscounting caused by cluster scratches. The reason is explained below.

[0073] First, a case where only the first correction process is performed will be described. As described above, the probability of occurrence of light emission crosstalk can be predicted based on the pixel structure of the photoelectric conversion element. However, due to manufacturing variations, the probability of occurrence of light emission crosstalk is not necessarily the same for all pixels. Therefore, in the first correction process, the effect of the correction process may not be sufficient due to the influence of manufacturing variations.

[0074] For example, in the case of a pixel whose output level difference is small compared to surrounding pixels, the effect that the pixel has on the surrounding pixels during the convolution operation is approximately the same as the effect that the surrounding pixels have on the pixel, so the effect of manufacturing variations is limited and the effect of the correction process is hardly reduced.

[0075] On the other hand, pixels that have a higher output level than the surrounding pixels, i.e., defective pixels that cause cluster defects, have a large impact on the surrounding pixels during convolution operations, so they are significantly affected by manufacturing variations and the effect of correction processing may not be sufficient.

[0076] That is, in pixels where the probability of occurrence of light emission crosstalk is low, the probability of the light emission crosstalk used during the convolution operation is higher than the light emission crosstalk that actually occurs, so blackening occurs in the pixels around the cluster flaw. On the other hand, in pixels where the probability of occurrence of light emission crosstalk is high, the probability of the light emission crosstalk used during the convolution operation is lower than the light emission crosstalk that actually occurs, so scratch residue occurs in the pixels around the cluster flaw.

[0077] Therefore, in the photoelectric conversion device of this embodiment, the miscounting caused by cluster defects that cannot be completely corrected by the first correction process is corrected by using a second correction process. In the second correction process, pixels with a large crosstalk estimation amount, i.e., pixels that are cluster defects, are corrected by an interpolation process. Therefore, compared to the convolution calculation used in the first correction process, it is possible to sufficiently reduce the influence of differences in the probability of light emission crosstalk caused by manufacturing variations on surrounding pixels.

[0078] Note that, when only the second correction process is performed, it is not possible to suppress the influence of miscounting of pixels other than defective pixels. In other words, by performing both the first correction process and the second correction process, it is possible to suppress the influence of miscounting and improve image quality.

[0079] In FIG. 8, in step S406, defect correction is performed on pixels in the third array data that are equal to or greater than the first threshold value. However, whether or not defect correction is performed may be determined based on the size of the second array data used in the first correction process, rather than on the third array data.

[0080] As mentioned above, pixels with a higher output level than the surrounding pixels have a large effect on the surrounding pixels due to the convolution operation in the first correction process. Therefore, if the size of the second array data used in the first correction process is N x M, the N x M pixels around the cluster scratch are pixels that may have black spots or residual scratches due to manufacturing variations.

[0081] Therefore, in step S405, it is preferable to set the pixels surrounding the defective pixel extracted in step S403 within a range of N or more by M or more as the target pixels for the defect correction. In other words, it is preferable to make the size of the range of pixels to be the target of the defect correction, which is the second correction target, larger than the size of the second array data.

[0082] <Embodiment 2> A photoelectric conversion device according to the second embodiment will be described in terms of a case where the photoelectric conversion element has a plurality of pixels with different spectral characteristics, which is a so-called color sensor, in contrast to the photoelectric conversion device according to the first embodiment. In the following, a Bayer array type color sensor of RGGB will be described as an example.

[0083] That is, for each pixel, one of the R, G, or B color filters is arranged in front of the pixel, and for example, in a given row, the color filters are arranged in the order R, G, R, G, and in the adjacent row, the color filters are arranged in the order G, B, G, B.

[0084] Fig. 11 is a flowchart of signal processing in the photoelectric conversion device according to the second embodiment. The operation of each step in the flowchart in Fig. 11 is performed by a computer in the photoelectric conversion device 300 executing a computer program stored in memory. The following mainly describes the parts that are different from the flowchart in Fig. 8. In step S501, similar to step S401, the image generation unit 303 generates a first image. Thereafter, similar to Fig. 8, the process branches into a first correction process and a second correction process.

[0085] After branching, the first correction process is performed, step S502 (convolution operation) and step S503 (subtraction) are the same as in Fig. 8. The probability of emission crosstalk is determined by the structure in the substrate on which the avalanche photodiodes are formed, and the influence of the color filters arranged on the substrate can be ignored. Therefore, the probability of emission crosstalk occurring is the same whether it is the monochrome sensor according to the first embodiment or the color sensor according to the second embodiment, and the same correction may be performed for the first correction process.

[0086] Similarly, steps S504 and S505 are the same as those in Fig. 8. When using the defect address stored in the memory in step S504, there is no difference between a monochrome sensor and a color sensor, and the influence of the color filter can be ignored for the third array data used in step S505.

[0087] However, the interpolation of the flaw in step S506 is different from step S406. Specifically, in step S506, an interpolation process is performed for each pixel with different spectral characteristics, that is, using signals from pixels of the same color (spectral characteristics). Specifically, for each pixel with different spectral characteristics, a median filter is applied, excluding pixels around the pixel to be interpolated whose third array data is equal to or greater than the first threshold value.

[0088] In the case of a color sensor with a Bayer array, pixels of the same color (spectral characteristics) are arranged every other pixel in each row and column, so if limited to pixels with the same spectral characteristics, the effect of cluster scratches is reduced more than in a monochrome sensor. Therefore, the kernel size of the median filter should be 3x3 or more.

[0089] In addition, in step S504, when the defective pixels are extracted from the first image without using the stored defect addresses, the defect extraction in step S504 in the second correction process is also different from step S404.

[0090] Fig. 12 is a diagram for explaining the second correction process according to the second embodiment, and as shown in Fig. 12, a median filter is applied to each pixel having a different spectral characteristic (pixels having the same spectral characteristic) of the first image to generate seventh images (four types). In Fig. 12, Ch1, Ch2, Ch3, and Ch4 respectively represent, for example, R, G, G, and B pixels.

[0091] Then, a pixel whose output difference between the first image and the seventh image is equal to or greater than the second threshold value is extracted as a defective pixel, and the difference between the first image and the seventh image is set as the output level of the defective pixel. Note that, although the kernel size of the median filter is 3x3 in FIG. 12, it may be larger than that. In this way, the defective pixel may be extracted by comparing the output of each pixel with the output of surrounding pixels for each pixel having different spectral characteristics using the first image.

[0092] Similarly, when steps S504 and S505 are integrated, they are different from steps S404 and S405. Specifically, as in Fig. 12, a seventh image is generated by applying a median filter to each pixel having a different spectral characteristic, and an eighth image is generated by subtracting the seventh image from the first image.

[0093] Then, the third array data is obtained by calculating the product of the signal level of each pixel of the eighth image and the first array data. Furthermore, for each color, a process is performed in which the output values ​​of pixels whose third array data are equal to or greater than the first threshold value are interpolated with the output values ​​of a plurality of pixels whose third array data are less than the first threshold value.

[0094] In the case of a color sensor, the second array data used in the convolution calculation in step S502 may be changed for each color. Similarly, the third array data used in step S504 may be changed for each color.

[0095] In the case of a color sensor, there is no difference in the probability of crosstalk between colors, but since the signal level changes for each pixel of a different color depending on the color of the subject, there are colors that are susceptible to the effects of erroneous counting due to crosstalk.

[0096] For example, in the case of a color sensor with photoelectric conversion elements in a Bayer array of RGGB, for a typical subject, the brightness of the B pixel is the lowest and the brightness of the G pixel is the highest. Therefore, the B pixel is most susceptible to miscounting due to crosstalk, and the G pixel is least susceptible.

[0097] The larger the size of the second array data, the more the influence of counting errors due to crosstalk can be reduced, but the larger the size of the second array data, the more susceptible it is to the difference in the probability of light emission crosstalk due to manufacturing variations. Therefore, it is desirable to set the second array data to the minimum size that can suppress the influence of counting errors due to crosstalk.

[0098] Therefore, the size of the second array data may be different for each pixel having different spectral characteristics. For example, in the case of a color sensor in which the photoelectric conversion element uses RGB color filters, it is preferable to make the size of the second array data used in step S502 larger for B pixels than for G pixels.

[0099] <Embodiment 3> The photoelectric conversion device according to the third embodiment differs from the photoelectric conversion device according to the first embodiment in the method of driving the photoelectric conversion element.

[0100] 13, the quench element 202 is composed of a MOS transistor, and the on / off of the quench element 202 (MOS transistor) is controlled by a control signal CLK connected to the gate of the MOS transistor. The control signal CLK for controlling the operation of the quench element is supplied from a control pulse generating unit 115 serving as a signal generating means.

[0101] 14 is a drive timing chart of the photoelectric conversion element according to embodiment 3. It shows the relationship between the control signal CLK of the quench element 202, the voltage of the node nodeA, the voltage of the node nodeB, and the output signal OUT of the selection circuit 212 in the photoelectric conversion element shown in FIG.

[0102] In the photoelectric conversion element of the third embodiment, when the control signal CLK is at a high level (e.g., 1 V), the drive voltage VH is not easily supplied to the APD, and when the control signal CLK is at a low level (e.g., 0 V), the drive voltage VH is supplied to the APD. When the control signal CLK is at a high level, the quench element 202 is turned off, and when the control signal CLK is at a low level, the quench element 202 is turned on.

[0103] The resistance value of the quench element 202 when the control signal CLK is at a high level is higher than the resistance value of the quench element 202 when the control signal CLK is at a low level. Therefore, when the control signal CLK is at a high level, a recharge operation is unlikely to occur even if avalanche multiplication occurs in the APD, and the voltage supplied to the APD becomes a voltage equal to or lower than the breakdown voltage of the APD. Therefore, the avalanche multiplication operation in the APD stops.

[0104] At time t1, the control signal CLK changes from high to low, turning on the quench element 202 and starting the recharge operation of the APD. This causes the voltage of the cathode of the APD to transition to a high level. Then, the voltage difference between the voltages applied to the anode and cathode of the APD becomes capable of avalanche multiplication.

[0105] The voltage of the cathode is the same as that of node A. Therefore, when the voltage of the cathode transitions from low level to high level, the voltage of node A becomes equal to or higher than the decision threshold at time t2. At this time, the pulse signal output from node B is inverted and goes from high level to low level.

[0106] When the recharge is completed, a voltage difference of the drive voltage VH-drive voltage VL is applied to the APD 201. After that, the control signal CLK goes high, and the quench element 202 is turned off.

[0107] Next, at time t3, when a photon is incident on the APD 201, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage of the cathode drops. That is, the voltage of the node A drops. When the voltage of the node A drops below the decision threshold while the voltage of the node A is dropping, the voltage of the node B goes from low level to high level.

[0108] That is, the portion of the output waveform at node A that exceeds the judgment threshold is shaped by the waveform shaping section 210 and output as a pulse signal at node B. Then, it is counted by the counter circuit, and the count value of the counter signal output from the counter circuit is increased by 1 LSB.

[0109] Between time t3 and time t4, photons are incident on the APD. However, since the quench element 202 is in the off state and the voltage applied to the APD 201 is not a voltage difference that allows avalanche multiplication, the voltage level of the node nodeA does not exceed the decision threshold.

[0110] At time t4, the control signal CLK changes from high to low, and the quench element 202 turns on. As a result, a current that compensates for the voltage drop from the drive voltage VL flows to the node nodeA, and the voltage of the node nodeA transitions to its original voltage level. At this time, the voltage of the node nodeA becomes equal to or higher than the determination threshold at time t5, so the pulse signal of the node nodeB is inverted and goes from high to low.

[0111] At time t6, the node A is stabilized to the original voltage level, and the control signal CLK goes from low to high. After this, the voltages of the nodes and signal lines change in response to the control signal CLK and the incidence of photons, as described from time t1 to time t6.

[0112] In this manner, by applying the control signal CLK to the quench element 202 to switch the quench element 202 on and off at a predetermined cycle, it is possible to control the recharge frequency of the APD.

[0113] If the control signal CLK is not used, there is a problem that the actual count value becomes smaller than the count value corresponding to the brightness of the incident light when the brightness is high. However, this problem can be solved by applying the control signal CLK to the quench element 202 to periodically switch the quench element 202 on and off.

[0114] However, when the recharge frequency of the APD is controlled by the control signal CLK, the relationship between the number of input signals and the number of output signals is nonlinear, but if the effects of light emission crosstalk are ignored, the relationship between the number of input signals and the number of output signals can be theoretically derived. Specifically, when the number of input signals is Nph, the number of output signals is Nct, the frequency of the control signal CLK (the inverse of the number of CLKs per unit time) is f, and the length of the exposure period is T, the following formula 1 is satisfied.

[0115]

number

[0116] In the photoelectric conversion device according to the third embodiment, a correction process is carried out that can simultaneously reduce the influence of the nonlinear response caused by the control signal CLK and the influence of erroneous counting caused by crosstalk, as will be described below.

[0117] Fig. 15 is a flowchart of signal processing in the photoelectric conversion device according to embodiment 3. Note that the operation of each step in the flowchart in Fig. 15 is performed by the computer in the photoelectric conversion device 300 executing a computer program stored in the memory.

[0118] The following description will focus on the differences from the flowchart in Fig. 8. In step S601, similar to step S401, the image processing unit generates a first image. Then, similar to Fig. 8, the process branches into a first correction process and a second correction process.

[0119] First, the first correction process will be described. In the photoelectric conversion device according to the third embodiment, the first correction process is further branched. In step S602, as described above, a convolution operation is performed between the first image and the second array data to generate a second image. As described above, the probability of occurrence of light emission crosstalk is predictable, so that miscounting due to light emission crosstalk can be predicted by performing a convolution operation.

[0120] In step S603, a nonlinearity correction process is performed on the first image to return the nonlinear response generated by the control signal CLK to a linear state (to correct linearity), and a ninth image is generated. Specifically, the number of input signals Nph is calculated from the number of output signals Nct so as to satisfy the following formula 2.

[0121]

number

[0122] Then, in step S604, the second image is subtracted from the ninth image to generate a third image. That is, the second image generated by convolving the first image with the second array data generated based on the first array data is subtracted from the ninth image generated by performing nonlinear correction processing on the first image to generate the third image.

[0123] As mentioned above, the second image is a signal representing a miscount caused by light emission crosstalk, so the third image is an image that restores the signal that would be obtained if no miscount caused by light emission crosstalk occurred.

[0124] In this way, by performing the processes in steps S602 to S604, it is possible to simultaneously reduce the influence of the nonlinear response caused by the control signal CLK and the influence of erroneous counting caused by crosstalk.

[0125] Fig. 16 is a diagram for explaining the first correction process according to embodiment 3. In Fig. 16, (A) shows the first image, (B) shows the second array data, (C) shows the second image, (D) shows the ninth image, and (E) shows the third image.

[0126] The second correction process in the third embodiment is similar to the correction process of the photoelectric conversion device in the first embodiment. That is, defective pixels are extracted in step S605, and third array data is obtained in step S606. After that, in step S607, defect correction is performed on pixels in the third array data that are equal to or greater than the first threshold value, and a fourth image is generated.

[0127] In the third embodiment, by performing the above-described signal processing, it is possible to suppress the influence of the nonlinear response caused by the control signal CLK, the influence of erroneous counting caused by crosstalk, and the influence of cluster defects.

[0128] <Embodiment 4> The photoelectric conversion device according to the fourth embodiment differs from the photoelectric conversion device according to the third embodiment in that the signal processing unit 302 includes a gain adjustment unit (not shown) that serves as a gain adjustment means for applying a gain to the first image. In general, in a photoelectric conversion device that acquires an image, the exposure time, the F-number of the imaging optical system, and the gain of the signal processing circuit (hereinafter referred to as signal gain) are changed depending on the brightness of the subject, and the image is captured so that the brightness of the subject is appropriate.

[0129] In the photoelectric conversion device according to the fourth embodiment, when the signal gain value is changed to adjust the brightness, the parameters of the first correction process and the second correction process are changed accordingly.

[0130] Fig. 17 is a flowchart of signal processing in the photoelectric conversion device according to embodiment 4. Note that the operation of each step in the flowchart in Fig. 17 is performed by the computer in the photoelectric conversion device 300 executing a computer program stored in the memory.

[0131] The following description will focus on the differences from the flowchart in Fig. 16. In step S701 in Fig. 17, a first image is generated by the image processing unit, similar to step S601. Then, the process branches into a first correction process and a second correction process, similar to Fig. 16.

[0132] In step S702, if the signal gain of the first image is adjusted by a gain adjustment unit (not shown), a convolution operation of the second array data is performed on the gain-adjusted first image to generate a second image.

[0133] The second array data generated based on the emission crosstalk probability often has a decimal value. Since the image output is generally an integer, the effect of quantization error can be suppressed and correction accuracy can be improved by adjusting the gain of the first image during the convolution operation. Note that the same signal gain applied in step S702 (hereinafter referred to as the first gain) is also used in steps S703 and S707.

[0134] In step S703, the first image is similarly multiplied by the first gain, and nonlinear correction processing is performed to return the nonlinear response generated by the control signal CLK to a linear state, thereby generating a ninth image. Since Equation 2 is an operation including logarithms, decimal precision is required. Therefore, in the fourth embodiment, when performing nonlinear correction processing to return the nonlinear response to a linear state, the first gain is multiplied by the image, thereby suppressing the influence of quantization errors and improving correction precision.

[0135] In step S703, a processing circuit for calculating Equation 2 may be implemented to determine the number of input signals Nph from the number of output signals Nct, or a lookup table having the characteristics of Equation 2 may be used. When using a lookup table, it is preferable to use a lookup table that combines the first gain with a process for restoring to linearity, since this can reduce quantization errors.

[0136] Steps S704 and S705 are the same as steps S604 and S605, respectively. That is, in step S704, the second image is subtracted from the ninth image to generate a third image, and in step S705, defective pixels are extracted.

[0137] In step S706, the first image is multiplied by the first gain and the product of the first image and the first array data is calculated to obtain third array data. As in step S702, by multiplying the image by the gain in advance during the convolution operation, the effect of quantization error can be suppressed and correction accuracy can be improved. After that, in step S707, defect correction is performed on pixels in the third array data that are equal to or greater than the first threshold value to generate a fourth image.

[0138] In this way, in the case of a photoelectric conversion device having a gain adjustment unit that adjusts the signal gain, it is desirable to change the parameters of the second correction process depending on the value of the signal gain. The darker the subject is and the larger the value of the signal gain applied to adjust the brightness is, the more visually noticeable the effect of cluster scratches is.

[0139] Therefore, it is desirable to decrease the first threshold value as the signal gain value increases, and increase the proportion of pixels to be corrected in step S707. In other words, it is desirable to strengthen the second correction process as the signal gain value applied in the gain adjustment unit increases.

[0140] Similarly, it is preferable to change the parameters of the first correction process according to the value of the signal gain.When the response is nonlinear due to the control signal CLK, the smaller the number of output signals Nct, the greater the effect of crosstalk.

[0141] Therefore, the darker the subject and the larger the signal gain value applied to adjust the brightness, the larger the size of the second array data used in the convolution operation and the larger the coefficients of the second array data are. In other words, the larger the signal gain value applied in the gain adjustment unit, the stronger the first correction process is.

[0142] Furthermore, it is desirable to simultaneously change both the parameters of the first correction process and the parameters of the second correction process by the signal gain. If the size of the second array data is increased or each coefficient of the second array data is increased, it becomes more susceptible to the influence of differences in the probability of light emission crosstalk due to cluster scratches and manufacturing variations.

[0143] Therefore, it is desirable to reduce the effect of cluster defects by changing the size and coefficients of the second array data and the first threshold value. In other words, it is desirable to strengthen both the first correction process and the second correction process as the signal gain value increases.

[0144] Also, the value of the frequency f of the control signal CLK may be changed according to the value of the signal gain. As can be seen from Equation 1, the saturation level of Nct is proportional to f×T, so if the length of exposure time T is the same, the frequency f of the control signal CLK and the saturation level of Nct are proportional. Therefore, it is desirable to increase the value of the signal gain and increase the brightness as the frequency f of the control signal CLK is lower.

[0145] Since Equation 2 depends on the frequency f of the control signal CLK, it is desirable to change the nonlinearity correction process in step S703 according to the frequency f. When using a lookup table having the characteristics of Equation 2, it is sufficient to use a different lookup table according to the frequency f of the control signal CLK.

[0146] As can be seen from Equation 1, if the exposure time length T is the same, the smaller the frequency f of the control signal CLK, the more nonlinear the response becomes, and therefore the greater the effect of crosstalk. Therefore, the lower the frequency f, the larger the size of the second array data used in the convolution operation, or the larger the coefficients of the second array data. In other words, the lower the frequency f, the stronger the first correction process is.

[0147] <Embodiment 5> 18 is a functional block diagram of a photoelectric conversion system according to embodiment 5 using the photoelectric conversion device of embodiments 1 to 4. The photoelectric conversion system 800 includes a photoelectric conversion device 300 having a photoelectric conversion element 100, a control unit 801, a storage unit 802, and a communication unit 803.

[0148] The photoelectric conversion element 100 captures an optical image formed by an imaging optical system 301. A signal read from the photoelectric conversion element 100 is subjected to image generation, a first correction process, and a second correction process in a signal processing unit 302.

[0149] Furthermore, the signal processor 302 performs processes such as black level correction, gamma curve adjustment, noise reduction, and data compression to generate a final image. In addition, if the photoelectric conversion element 100 has an on-chip RGB color filter, it is desirable for the signal processor 302 to perform processes such as white balance correction and color conversion.

[0150] The control unit 801 has a built-in CPU (not shown) as a computer, and functions as a control means for controlling the operation of each part of the entire photoelectric conversion system 800 based on a computer program stored in a memory (not shown) as a storage medium. The control unit 801 also controls the length of the exposure period of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK via a control pulse generation unit of the photoelectric conversion element 100.

[0151] The storage unit 802 includes a recording medium such as a memory card, a hard disk, etc. The communication unit 804 includes a wireless or wired interface, and outputs a generated image to the outside of the photoelectric conversion system 800 and receives a signal from the outside.

[0152] Although the present invention has been described in detail based on the preferred embodiment, the present invention is not limited to the above embodiment, and various modifications are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. Note that this embodiment includes the following combinations.

[0153] (Configuration 1) A photoelectric conversion element constituted by an avalanche photodiode for photoelectrically converting an optical image, an image generating means for generating a first image based on a signal acquired by the photoelectric conversion element, and an acquiring means for acquiring first characteristic information of the photoelectric conversion element; A photoelectric conversion device comprising: a first correction processing means for performing a first correction processing for correcting linearity of the first image using the first characteristic information; and a second correction processing means for performing a second correction processing for pixel interpolation based on both the first characteristic information and information of the first image.

[0154] (Configuration 2) The photoelectric conversion device described in Configuration 1, wherein the first characteristic information is two-dimensional array data, and one-dimensional data corresponding to at least one row or one column of the two-dimensional array data has a peak value in the center.

[0155] (Configuration 3) The photoelectric conversion device according to configuration 2, wherein the one-dimensional data has a distribution that monotonically decreases from the peak value toward the data end.

[0156] (Configuration 4) A photoelectric conversion device according to configuration 2 or 3, characterized in that other one-dimensional data that shares the peak value of the one-dimensional data and is arranged in a direction intersecting the one-dimensional data has a distribution that monotonically decreases from the peak value of the other one-dimensional data toward the end of the data.

[0157] (Configuration 5) The photoelectric conversion device according to any one of configurations 2 to 4, wherein the two-dimensional array data has an odd number of rows and columns and is symmetrical above, below, left, and right about the center.

[0158] (Configuration 6) A photoelectric conversion device described in any one of configurations 1 to 5, characterized in that the first correction process includes a process of generating a second image by convolving second array data created based on the first characteristic information with the first image, and a process of generating a third image by subtracting the second image from the first image.

[0159] (Configuration 7) The photoelectric conversion device according to configuration 6, wherein the size of the range of pixels that is the target of the second correction process is larger than the size of the second array data.

[0160] (Configuration 8) A photoelectric conversion device described in Configuration 6 or 7, characterized in that the second correction process includes a process of generating third array data based on the first characteristic information and the output values ​​of the first image, and a process of correcting the output values ​​of the third image based on the third array data.

[0161] (Configuration 9) The photoelectric conversion device described in Configuration 8, characterized in that the third array data is generated by multiplying the first characteristic information by the output value of the first image, and the output values ​​of pixels whose values ​​of the third array data are greater than or equal to a first threshold value are interpolated with the output values ​​of pixels whose values ​​of the third array data are less than the first threshold value.

[0162] (Configuration 10) The photoelectric conversion device according to any one of configurations 1 to 9, wherein the second correction process further includes a process of extracting defective pixels from the first image.

[0163] (Configuration 11) The photoelectric conversion device according to configuration 10, wherein the process of extracting the defective pixels is performed based on pre-stored address data.

[0164] (Configuration 12) The photoelectric conversion device according to configuration 10, wherein the process of extracting the defective pixels extracts the defective pixels by comparing the output of each pixel of the first image with the output of surrounding pixels.

[0165] (Configuration 13) The photoelectric conversion device according to any one of configurations 1 to 12, wherein the photoelectric conversion element has a plurality of pixels having different spectral characteristics.

[0166] (Configuration 14) The photoelectric conversion device described in Configuration 13, characterized in that the second correction process generates third array data by multiplying the first characteristic information by the output value of the first image, and for each pixel having different spectral characteristics, interpolates the output value of a pixel whose value of the third array data is greater than or equal to a first threshold value with the output value of a pixel whose value of the third array data is less than the first threshold value.

[0167] (Configuration 15) The photoelectric conversion device described in configuration 13 or 14, characterized in that the second correction process includes a process of extracting defective pixels by using the first image to compare the output of each pixel with the output of surrounding pixels for each pixel having different spectral characteristics.

[0168] (Configuration 16) The first correction process includes a process of generating a second image by convolving second array data created based on the first characteristic information with the first image; 16. The photoelectric conversion device according to any one of configurations 13 to 15, wherein the size of the second array data is different for each pixel having different spectral characteristics.

[0169] (Configuration 17) The photoelectric conversion device described in any one of configurations 1 to 16, characterized in that the photoelectric conversion element has a quench element connected to one of the anode and cathode of the avalanche photodiode and a power line, and a signal generating means for supplying a control signal for controlling the operation of the quench element.

[0170] (Configuration 18) The first correction process is carried out by the following formula, where f is the frequency of the control signal, T is the length of the exposure time, Nph is the number of input signals, and Nct is the number of output signals. 18. The photoelectric conversion device according to claim 17, wherein nonlinearity is corrected so as to satisfy TIFF2024048797000004.tif22159.

[0171] (Configuration 19) The photoelectric conversion device described in Configuration 18, characterized in that the first correction process generates a third image by subtracting a second image generated by convolving second array data created based on the first characteristic information with the first image from a ninth image generated by performing nonlinear correction processing on the first image.

[0172] (Configuration 20) The photoelectric conversion device according to any one of configurations 1 to 19, further comprising a gain adjustment means for applying a gain to the first image.

[0173] (Configuration 21) The photoelectric conversion device according to configuration 20, wherein the first correction process is strengthened as the value of the gain applied by the gain adjustment means increases.

[0174] (Configuration 22) The photoelectric conversion device according to configuration 20, wherein the second correction process is strengthened as the value of the gain applied by the gain adjustment means increases.

[0175] (Configuration 23) The photoelectric conversion device according to configuration 20, wherein the larger the value of the gain applied by the gain adjustment means is, the stronger both of the first correction process and the second correction process are made.

[0176] (Configuration 24) A photoelectric conversion device described in any one of configurations 20 to 23, characterized in that the photoelectric conversion element has a quench element connected to one of the anode and cathode nodes of the avalanche photodiode and a power line, and a signal generating means for supplying a control signal for controlling the operation of the quench element, and the lower the frequency of the control signal, the larger the value of the gain applied by the gain adjustment means.

[0177] (Configuration 25) The photoelectric conversion device according to any one of configurations 17 to 24, wherein the first correction process is strengthened as the frequency of the control signal is lower.

[0178] (Configuration 26) The photoelectric conversion device according to any one of configurations 1 to 25, wherein the first characteristic is a crosstalk characteristic between pixels caused by an avalanche light emission phenomenon.

[0179] (Configuration 27) The photoelectric conversion device according to any one of configurations 1 to 26, wherein the second correction process is a defect correction process for interpolating a signal of a defective pixel with surrounding pixel signals.

[0180] (Configuration 28) A photoelectric conversion device comprising: a photoelectric conversion element composed of an avalanche photodiode for photoelectrically converting an optical image; an image generating means for generating a first image based on a signal acquired by the photoelectric conversion element; an acquisition means for acquiring first characteristic information relating to crosstalk characteristics between pixels caused by the avalanche light emission phenomenon of the photoelectric conversion element; and a first correction processing means for performing a first correction processing in which a result of a predetermined calculation using the information and the first image is subtracted from the first image.

[0181] (Method) A control method for controlling a photoelectric conversion device having a photoelectric conversion element composed of an avalanche photodiode for photoelectrically converting an optical image, the control method comprising: an image generation step of generating a first image based on a signal acquired by the photoelectric conversion element; an acquisition step of acquiring first characteristic information of the photoelectric conversion element; a first correction processing step of performing a first correction processing on the first image using the first characteristic information; and a second correction processing step of performing a second correction processing based on both the first characteristic information and information on the first image.

[0182] (Program) A computer program for controlling each means of the photoelectric conversion device according to any one of configurations 1 to 28 by a computer.

[0183] A computer program for implementing the functions of the above-described embodiment as part or all of the control in this embodiment may be supplied to the photoelectric conversion device via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the photoelectric conversion device may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0184] 100: Photoelectric conversion element 11: Sensor board 12: Pixel area 21: Circuit board 22: Circuit area 101: Pixel 102: Photoelectric conversion unit 103: Signal processing circuit 110: Vertical scanning circuit 111: Horizontal scanning circuit 112: Readout circuit 113: Vertical signal line 114: Output circuit 115: Control pulse generation unit 201: Avalanche photodiode 202: Quench element 210: Waveform shaping section 211: Counter circuit 212: Selection circuit 213: Drive wire 300: Photoelectric conversion device 301: Imaging optical system 302: Signal processing unit 303: Image generation unit 304: First correction processing unit 305: Second correction processing unit 306: Storage section 800: Photoelectric conversion system 801: Control unit 802: Storage section 803: Communications Department

Claims

1. a photoelectric conversion element configured with an avalanche photodiode for photoelectrically converting an optical image; an image generating means for generating a first image based on a signal acquired by the photoelectric conversion element; an acquisition means for acquiring first characteristic information related to crosstalk between pixels of the photoelectric conversion element; a first correction processing means for performing a first correction process for correcting linearity of the first image by using the first characteristic information; and second correction processing means for performing a second correction process for pixel interpolation based on both the first characteristic information and the information of the first image.

2. The photoelectric conversion device according to claim 1, characterized in that the first characteristic information is two-dimensional array data, and one-dimensional data corresponding to at least one row or one column of the two-dimensional array data has a peak value in the center.

3. 3. The photoelectric conversion device according to claim 2, wherein the one-dimensional data has a distribution in which the data monotonically decreases from the peak value toward an end of the data.

4. The photoelectric conversion device according to claim 2, characterized in that other one-dimensional data that shares the peak value of the one-dimensional data and is arranged in a direction intersecting the one-dimensional data has a distribution that monotonically decreases from the peak value of the other one-dimensional data toward the end of the data.

5. 3. The photoelectric conversion device according to claim 2, wherein the two-dimensional array data has an odd number of rows and columns and is symmetrical above, below, left and right with respect to the center.

6. The photoelectric conversion device according to claim 1, characterized in that the first correction process includes a process of generating a second image by convolving second array data created based on the first characteristic information onto the first image, and a process of generating a third image by subtracting the second image from the first image.

7. 7. The photoelectric conversion device according to claim 6, wherein a size of a range of pixels that is the object of the second correction process is larger than a size of the second array data.

8. The photoelectric conversion device according to claim 6, characterized in that the second correction process includes a process of generating third array data based on the first characteristic information and the output values ​​of the first image, and a process of correcting the output values ​​of the third image based on the third array data.

9. The photoelectric conversion device described in claim 8, characterized in that the third array data is generated by multiplying the first characteristic information by the output value of the first image, and the output values ​​of pixels whose values ​​of the third array data are greater than or equal to a first threshold value are interpolated with the output values ​​of pixels whose values ​​of the third array data are less than the first threshold value.

10. 2. The photoelectric conversion device according to claim 1, wherein the second correction process further includes a process of extracting defective pixels from the first image.

11. 11. The photoelectric conversion device according to claim 10, wherein the process of extracting the defective pixels is performed based on pre-stored address data.

12. 11. The photoelectric conversion device according to claim 10, wherein the process of extracting the defective pixels extracts the defective pixels by comparing an output of each pixel of the first image with an output of a surrounding pixel.

13. The photoelectric conversion device according to claim 1 , wherein the photoelectric conversion element has a plurality of pixels having different spectral characteristics.

14. The photoelectric conversion device described in claim 13, characterized in that the second correction process generates third array data by multiplying the first characteristic information by an output value of the first image, and for each pixel having different spectral characteristics, interpolates the output value of a pixel whose value of the third array data is equal to or greater than a first threshold value with the output value of a pixel whose value of the third array data is less than the first threshold value.

15. The photoelectric conversion device according to claim 13, characterized in that the second correction process includes a process of extracting defective pixels by using the first image to compare the output of each pixel with the output of surrounding pixels for each pixel having different spectral characteristics.

16. the first correction process includes a process of generating a second image by convolving second array data created based on the first characteristic information with the first image, 14. The photoelectric conversion device according to claim 13, wherein a size of the second array data is different for each pixel having a different spectral characteristic.

17. The photoelectric conversion device according to claim 1, characterized in that the photoelectric conversion element includes a quench element connected to one of the anode and cathode nodes of the avalanche photodiode and a power supply line, and a signal generating means for supplying a control signal for controlling the operation of the quench element.

18. The first correction process is carried out by the following formula, where f is the frequency of the control signal, T is the length of the exposure time, Nph is the number of input signals, and Nct is the number of output signals.

18. The photoelectric conversion device according to claim 17, wherein nonlinearity is corrected so as to satisfy the following:

19. The photoelectric conversion device according to claim 18, characterized in that the first correction process generates a third image by subtracting a second image generated by convolving second array data created based on the first characteristic information with the first image from a ninth image generated by performing nonlinear correction processing on the first image.

20. 2. The photoelectric conversion device according to claim 1, further comprising a gain adjustment unit that applies a gain to the first image.

21. 21. The photoelectric conversion device according to claim 20, wherein the first correction process is strengthened as the value of the gain applied by the gain adjustment means increases.

22. 21. The photoelectric conversion device according to claim 20, wherein the second correction process is strengthened as the value of the gain applied by the gain adjustment means increases.

23. 21. The photoelectric conversion device according to claim 20, wherein the larger the value of the gain applied by the gain adjustment means is, the stronger both of the first correction process and the second correction process are made.

24. The photoelectric conversion device described in claim 20, characterized in that the photoelectric conversion element has a quench element connected to one of the anode and cathode nodes of the avalanche photodiode and a power line, and a signal generating means for supplying a control signal for controlling the operation of the quench element, and the lower the frequency of the control signal, the larger the value of the gain applied by the gain adjustment means.

25. 18. The photoelectric conversion device according to claim 17, wherein the first correction process is strengthened as the frequency of the control signal decreases.

26. 2. The photoelectric conversion device according to claim 1, wherein the first characteristic information relates to a crosstalk characteristic between pixels caused by an avalanche light emission phenomenon.

27. 2. The photoelectric conversion device according to claim 1, wherein the second correction process is a defect correction process for interpolating a signal of a pixel having a defect with signals of surrounding pixels.

28. a photoelectric conversion element configured with an avalanche photodiode for photoelectrically converting an optical image; an image generating means for generating a first image based on a signal acquired by the photoelectric conversion element; an acquisition means for acquiring first characteristic information relating to a crosstalk characteristic between pixels caused by an avalanche light emission phenomenon of the photoelectric conversion element; a first correction processing means for performing a first correction process in which a result of a predetermined calculation using the first characteristic information and the first image is subtracted from the first image.

29. A method for controlling a photoelectric conversion device having a photoelectric conversion element configured with an avalanche photodiode for photoelectrically converting an optical image, comprising: an image generating step of generating a first image based on a signal acquired by the photoelectric conversion element; acquiring first characteristic information of the photoelectric conversion element; a first correction processing step of performing a first correction processing for correcting linearity of the first image by using the first characteristic information; a second correction processing step of performing a second correction processing for pixel interpolation based on both the first characteristic information and information of the first image.

30. A computer program for controlling each means of the photoelectric conversion device according to any one of claims 1 to 28 by a computer.

Images