Processing device, photoelectric conversion device, processing method and program

Abstract

To suppress degradation in image quality while correcting defective pixels due to different various causes.SOLUTION: There is provided a processing device for generating and correcting an image based on a count value outputted by a photoelectric conversion element that has an avalanche photodiode, converts light from a subject into an electrical signal, counts the electrical signal, and outputs the count value. The processing device includes image generation means for generating a first image based on the count value, correction processing means for executing correction processing to correct the first image. The correction processing means executes the correction processing including: first correction processing for a first pixel including a pixel in a surrounding area of a first type of defective pixel in which miscounting occurs due to a first cause; and second correction processing different from the first correction processing for a second pixel including a pixel in a surrounding area of a second type of defective pixel in which miscounting occurs due to a second cause different from the first cause.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a processing device, a photoelectric conversion device, a processing method, and a program. [Background technology]

[0002] BACKGROUND ART In recent years, photoelectric conversion devices have become known that count the number of photons incident on an avalanche photodiode (APD) and output the count value from a pixel as a photoelectrically converted digital signal.

[0003] It is known that a phenomenon called avalanche emission occurs in photoelectric conversion devices that include APDs. When avalanche emission occurs, the generated secondary electrons enter adjacent pixels, increasing the count values ​​of the adjacent pixels and causing miscounts. In particular, if the pixel in question is defective, crosstalk caused by the avalanche emission phenomenon can cause the defect to become a cluster defect that spans multiple pixels.

[0004] Patent Document 1 discloses a method for correcting cluster defects caused by crosstalk by using a matrix of the probability of light emission crosstalk caused by avalanche light emission as a correction array. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2023-118661 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in a photoelectric conversion device having an APD, defective pixels due to factors other than light-emission crosstalk may occur. If the same correction method as for light-emission crosstalk is applied to defective pixels due to such different factors, the image quality may be degraded by the correction.

[0007] Therefore, the present invention provides a processing device, a photoelectric conversion device, a processing method, and a program that correct defective pixels caused by a plurality of different factors in a device having an APD while suppressing degradation of image quality. [Means for solving the problem]

[0008] In order to solve this problem, for example, the processing apparatus of the present invention has the following configuration: A processing device that generates and corrects an image based on a count value output by a photoelectric conversion element having an avalanche photodiode that converts light from a subject into an electrical signal, counts the signal, and outputs the count value, an image generating means for generating a first image based on the count value; a correction processing means for executing a correction process to correct the first image; and The correction processing means a first correction process for first pixels including pixels surrounding a first type of defective pixel that causes a miscount due to a first cause; a second correction process different from the first correction process for second pixels including pixels surrounding a second type of defective pixel that causes a miscount due to a second cause different from the first cause; The correction process including the steps of: [Effects of the Invention]

[0009] According to the present invention, in an apparatus having an APD, it is possible to correct defective pixels caused by a plurality of different factors while suppressing degradation of image quality. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is an exploded perspective view showing an example of the overall configuration of a photoelectric conversion element according to an embodiment. [Figure 2] FIG. 2 is a schematic plan view showing a sensor substrate according to the embodiment. [Figure 3] FIG. 2 is a diagram showing an example of the configuration of a circuit board according to an embodiment. [Figure 4] FIG. 2 is a diagram showing an equivalent circuit of a photoelectric conversion unit and a signal processing circuit of a pixel according to an embodiment. [Figure 5] FIG. 2 is a timing chart schematically illustrating the relationship between the operation of an APD and an output signal. [Figure 6] FIG. 1 is a block diagram showing the configuration of a photoelectric conversion device according to a first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of first sequence data. [Figure 8] FIG. 4 is a flowchart of signal processing in the photoelectric conversion device according to the first embodiment. [Figure 9] 5A to 5C are diagrams showing changes in an image in the correction process of the first embodiment. [Figure 10] FIG. 10 is a diagram showing the luminance distribution around a defective pixel before signal processing. [Figure 11] 10A and 10B are diagrams illustrating a state during signal processing in a comparative example. [Figure 12] 10A and 10B are diagrams illustrating the results of signal processing in a comparative example. [Figure 13] 6A and 6B are diagrams showing changes in an image (first type of defective pixel) in another correction process according to the first embodiment. [Figure 14] 8A and 8B are diagrams showing changes in an image (a second type of defective pixel) in another correction process according to the first embodiment. [Figure 15] FIG. 4 is a flowchart showing a defective pixel extraction process according to the first embodiment. [Figure 16] 4A and 4B are diagrams illustrating the periphery of a pixel of interest, explaining the details of the defective pixel extraction process according to the first embodiment; [Figure 17] FIG. 10 is a diagram showing an equivalent circuit of a photoelectric conversion unit and a signal processing circuit of a pixel according to a second embodiment. [Figure 18] FIG. 10 is a timing chart of the photoelectric conversion element according to the second embodiment. [Figure 19] 10 is a flowchart of signal processing in a photoelectric conversion device according to a second embodiment. [Figure 20] FIG. 10 is a block diagram showing the overall configuration of a photoelectric conversion system according to a third embodiment. [Figure 21] FIG. 10 is a block diagram showing the hardware configuration of a control unit according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] <Photoelectric conversion element> 1 is an exploded perspective view showing an example of the overall configuration of a photoelectric conversion element according to an embodiment of the present invention, and the photoelectric conversion element will be described with reference to FIG.

[0013] As shown in FIG. 1, the photoelectric conversion element 100 has a sensor substrate 11 and a circuit board 21. The sensor substrate 11 and the circuit board 21 are stacked and electrically connected to each other. That is, the photoelectric conversion element 100 has a stacked structure. The photoelectric conversion element 100 is not limited to the above-described configuration. For example, the photoelectric conversion element 100 may have a so-called non-stacked structure in which the components included in the sensor substrate and the components included in the circuit board are arranged on a common semiconductor layer. The sensor substrate 11 includes a pixel region 12 including a plurality of pixels. The circuit board 21 includes a circuit region 22 that processes signals detected by the pixels in the pixel region 12.

[0014] <Sensor board> 2 is a schematic plan view showing the configuration of a sensor substrate 11 according to an embodiment. A pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally (also referred to as a matrix) across a plurality of rows and columns. Each pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter referred to as APD). The number of rows and columns of the pixels 101 in the pixel region 12 is not particularly limited.

[0015] <Circuit board> 3 is a diagram showing an example of the configuration of a circuit board 21 according to an embodiment. The circuit board 21 includes a signal processing circuit 103 that processes charges photoelectrically converted by the photoelectric conversion unit 102 shown in FIG. 2, a vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, signal lines 113, and a control pulse generation unit 115.

[0016] The signal processing circuit 103 acquires and processes the electrical signals output from the photoelectric conversion units 102 of the pixels. The signal processing circuit 103 includes a counter associated with each pixel, a memory for storing digital values, and the like. The signal processing circuit 103 counts the number of photons incident on the pixel and stores the count value in the memory. The signal processing circuit 103 outputs the count value based on a control pulse, which will be described later.

[0017] The vertical scanning circuit 110 receives a control pulse supplied from a control pulse generating unit 115 and supplies the control pulse to the signal processing circuit 103 of each pixel. The vertical scanning circuit 110 has logic circuits such as a shift register and an address decoder.

[0018] The horizontal scanning circuit 111 supplies control pulses for sequentially selecting each column to the signal processing circuit 103 in order to read out the signals from the memory of each pixel where the digital signals are stored. The control pulses supplied by the horizontal scanning circuit 111 are pulses for reading out pixel signals including the count values ​​of each pixel stored in the memory of the signal processing circuit 103.

[0019] The readout circuit 112 reads out pixel signals, including count values ​​output by the signal processing circuit 103, for each column via a signal line 113 based on control pulses generated by the vertical scanning circuit 110. The readout circuit 112 includes a shift register and an address decoder, connecting multiple rows as a single unit. Therefore, the readout circuit 112 achieves high-speed readout by simultaneously reading out pixel signals from multiple rows. In particular, in an imaging device that digitally counts the number of photons incident on the APD of each pixel and outputs the count value as a photoelectrically converted digital signal from the pixel, the operation of the counter circuit that digitally counts the number of photons takes time. Therefore, high-speed pixel signal readout is achieved by simultaneously reading out signals from pixels in multiple rows. In this embodiment, the readout circuit 112 simultaneously reads out pixel signals from the pixels in the first row and the pixels in the second row, for example, for each column.

[0020] A pixel signal including a count value and the like is output to the signal line 113 from the signal processing circuit 103 of the pixel at which the column selected by the horizontal scanning circuit 111 and one or more rows selected by the vertical scanning circuit 110 intersect.

[0021] The output circuit 114 outputs the pixel signal output via the signal line 113 and the readout circuit 112 to the outside of the photoelectric conversion element 100 .

[0022] The control pulse generation unit 115 controls the photoelectric conversion element 100. Specifically, the control pulse generation unit 115 supplies control pulses to the vertical scanning circuit 110 and the horizontal scanning circuit 111 in order to selectively read out pixel signals from each pixel.

[0023] <Connection between sensor board and circuit board> 2 and 3, a plurality of signal processing circuits 103 are arranged in an area overlapping with the pixel area 12 in a planar view. The sensor substrate 11 has the pixel area 12 and a non-pixel area around the pixel area 12. The non-pixel area is an area between an end of the sensor substrate 11 and an end of the pixel area 12. The vertical scanning circuit 110, the horizontal scanning circuit 111, the readout circuit 112, the output circuit 114, and the control pulse generating unit 115 are arranged in the non-pixel area in a planar view.

[0024] The arrangement of the vertical scanning circuit 110, horizontal scanning circuit 111, readout circuit 112, signal lines 113, and output circuit 114 is not limited to that shown in FIG. 3. For example, the signal lines 113 may be arranged extending in the row direction, and the readout circuits 112 may be arranged at the ends of the signal lines 113. Furthermore, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit 102, and one signal processing circuit 103 may be shared by multiple photoelectric conversion units 102. In this case, a configuration in which signal processing is performed sequentially may be used.

[0025] <Pixel equivalent circuit> 4 is a diagram showing an equivalent circuit of the photoelectric conversion unit 102 and the signal processing circuit 103 of the pixel 101. As shown in FIG.

[0026] The APD201 is an avalanche photodiode that generates charge pairs according to the incident light (number of photons) through photoelectric conversion. One of the two nodes of the APD201 is connected to a power supply line that supplies a drive voltage VL (first voltage). The other of the two nodes of the APD201 is connected to a power supply line that supplies a drive voltage VH (second voltage) that is higher than the drive voltage VL. In FIG. 4, one node of the APD201 is an anode, and the other node of the APD201 is a cathode. A reverse bias voltage is supplied to the anode and cathode of the APD201 so that the APD201 performs avalanche multiplication. With this voltage supplied, avalanche multiplication occurs in the APD201 due to charges generated by the incident light, generating an avalanche current.

[0027] When a reverse bias voltage is supplied, the APD 201 operates in either Geiger mode, where the voltage difference between the anode and cathode is greater than the breakdown voltage, or in linear mode, where the voltage difference between the anode and cathode is close to or less than the breakdown voltage. An APD operating in Geiger mode is called a SPAD (Single Photon Avalanche Diode). In the case of a SPAD, for example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V.

[0028] The signal processing circuit 103 has a quench element 202, a waveform shaping section 210, a counter circuit 211, and a selection circuit 212. It is sufficient that the signal processing circuit 103 has at least one of the waveform shaping section 210, the counter circuit 211, and the selection circuit 212.

[0029] The quench element 202 is connected to a power supply line to which a drive voltage VH is supplied and to one of the anode and cathode nodes of the APD 201. The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication. Therefore, the quench element 202 suppresses the voltage supplied to the APD 201 to suppress avalanche multiplication (quench operation). The quench element 202 returns the voltage supplied to the APD 201 to the drive voltage VH by flowing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

[0030] The waveform shaping unit 210 shapes the voltage change at the cathode of the APD 201 obtained when a photon is detected, and outputs a pulse signal. The waveform shaping unit 210 may be, for example, a single inverter circuit. Note that the waveform shaping unit 210 may also be configured with multiple inverter circuits connected in series, or may be any other circuit that has a waveform shaping effect.

[0031] The counter circuit 211 counts the pulse signals output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via a drive line 213, the counter circuit 211 outputs the held count value and resets the count value.

[0032] When a control pulse SEL is supplied from the vertical scanning circuit 110 in Fig. 3 via a drive line 214 (not shown in Fig. 3) in Fig. 4, the selection circuit 212 switches between electrical connection and disconnection between the counter circuit 211 and the signal line 113. The selection circuit 212 includes, for example, a buffer circuit for outputting a signal. An output signal OUT shown in Fig. 4 is an output signal from a pixel.

[0033] The electrical connections may be switched by disposing switches such as transistors between the quench element 202 and the APD 201 and between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the voltage VH or the voltage VL to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

[0034] <Circuit drive> FIG. 5 is a timing chart showing the relationship between the APD operation and the output signal. Node A indicates the input side of the waveform shaping unit 210. Node B indicates the output side of the waveform shaping unit 210. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at node A drops. As the voltage drop increases and the potential difference applied to the APD 201 decreases, avalanche multiplication in the APD 201 stops, as shown at time t2, and the voltage level at node A no longer drops below a certain value. After time t2, a current flows through node A from voltage VL to compensate for the voltage drop, causing the potential level to rise and reach VL. Thereafter, node A remains at its original potential level until time t3. At this time, the potential level at the time when the output waveform at node A exceeds a certain determination threshold is waveform-shaped by the waveform shaping section 210 and is output as a signal from node B.

[0035] The photoelectric conversion devices of the respective embodiments will be described below.

[0036] (First embodiment) <Photoelectric conversion device> 6 is a block diagram showing the configuration of a photoelectric conversion device 300 shown in the first embodiment. The photoelectric conversion device 300 includes a photoelectric conversion element 100 having an APD 201 described with reference to FIGS. 1 to 5, an imaging optical system 301, and a signal processing unit 302 that processes a signal acquired by the photoelectric conversion element 100.

[0037] The imaging optical system 301 includes a lens, a diaphragm, etc. The imaging optical system 301 guides light from a subject to the photoelectric conversion element 100.

[0038] The signal processing unit 302 is an example of a processing device, and includes an image generation unit 303, a correction processing unit 304, and a storage unit 305. The image generation unit 303 and the correction processing unit 304 may be realized by one or more circuits such as an ASIC (Application Specific Integrated Circuit) and a PLD (Programmable Logic Device) including an FPGA (Field Programmable Gate Array).

[0039] The image generating unit 303 generates a first image from the pixel signals acquired from the photoelectric conversion element 100 .

[0040] The correction processing unit 304 performs correction processing on the pixel values ​​(also called output values) of pixels around the defective pixel using information of the first array data based on characteristic information of the photoelectric conversion element 100.

[0041] The storage unit 305 has storage devices such as a RAM (Random Access Memory), a ROM (Read Only Memory), and storages such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive). The storage unit 305 stores a computer program for executing image generation and correction processing, image data, first array data based on characteristic information of the photoelectric conversion element 100, and the like.

[0042] As disclosed in Patent Document 1, when a pixel has an APD, avalanche light emission phenomenon causes erroneous counting in adjacent pixels, that is, crosstalk between adjacent pixels (hereinafter referred to as light emission crosstalk).

[0043] The effect of light-emission crosstalk on adjacent pixels is determined by the probability of light-emission crosstalk occurring. The probability of light-emission crosstalk occurring is determined by the pixel structure of the photoelectric conversion element, so the probability of light-emission crosstalk occurring can be predicted based on the pixel structure of the photoelectric conversion element.

[0044] That is, the characteristic information means the probability of occurrence of light emission crosstalk determined by the pixel structure of the photoelectric conversion element.

[0045] FIG. 7 is a diagram showing an example of the first array data. As shown in FIG. 7, the first array data associates pixel positions with the probability of miscounting due to the probability of light-emission crosstalk. Each block represents one pixel. The probability of miscounting is an example of a correction value. More specifically, if light-emission crosstalk occurs in a central pixel, the probability of miscounting of the central pixel is 100%, and the probability of miscounting of pixels adjacent to the central pixel on the left, right, top, and bottom (also called nearest neighboring pixels) is 1%. The first array data is stored in the memory unit 305. The correction processing unit 304 uses the first array data to correct pixels surrounding the defective pixel.

[0046] <Flowchart> FIG. 8 is a flowchart of signal processing according to the first embodiment. FIG. 9 is a diagram illustrating changes in an image during the correction process described above. FIG. 9(a) is a diagram of a first image. The upper first image of FIG. 9(a) shows a second type of defective pixel and its surrounding pixels. The lower first image of FIG. 9(a) shows a first type of defective pixel and its surrounding pixels. FIG. 9(b) is a diagram of second array data. FIG. 9(c) is a diagram illustrating a second image that is the result of a convolution operation performed on the first type of defective pixel. FIG. 9(d) is a diagram illustrating a second image that is the result of a convolution operation performed on the second type of defective pixel. FIG. 9(e) is a diagram illustrating a third image. The upper third image of FIG. 9(e) shows a second type of defective pixel and its surrounding pixels. The lower third image of FIG. 9(e) shows a first type of defective pixel and its surrounding pixels. The first signal processing will be described with reference to FIGS. 8 and 9.

[0047] First, in step S401, the image generating unit 303 generates a first image arranged in a two-dimensional frame shape from pixel signals including count values ​​acquired from the photoelectric conversion elements 100.

[0048] Next, in step S402, correction processing unit 304 performs defective pixel extraction processing on the first image generated by image generation unit 303, extracting the first type of defective pixels and the second type of defective pixels shown in FIG. 9(a). Specific details of the defective pixel extraction processing will be described later. The positions of the first type of defective pixels and the second type of defective pixels may be determined by acquiring a dark image in advance, extracting information about the defective pixels from the dark image, and storing this information in memory as address data. The defective pixel extraction processing here includes processing for extracting defective pixels and classifying the extracted defective pixels into first type defective pixels and second type defective pixels.

[0049] Next, in step S403, the correction processing unit 304 generates a second image using the first array data. Specifically, the correction processing unit 304 performs a convolution operation on the first image using second array data created based on the first array data, to generate the second image. As described above, the probability of occurrence of light emission crosstalk is predictable, so by performing a convolution operation using the second array data, it is possible to predict miscounting due to light emission crosstalk.

[0050] The second array data can be created by replacing the value of the center element (100%) of the first array data with zero (see FIG. 9(b)). By creating the second array data in this way, the correction processing unit 304 creates a second image that restores signals that represent miscounts caused by light emission crosstalk. However, the second array data may be different as long as it is based on the first array data.

[0051] The correction processing unit 304 generates second pixels by performing a convolution operation on the first image using the array data obtained by multiplying the second array data by a weighting factor. The correction processing unit 304 differentiates the weighting factor K2 of the correction process around the second type of defective pixel from the weighting factor K1 of the correction process around the first type of defective pixel (see FIG. 9(b)). For example, the correction processing unit 304 sets the weighting factor K2 of the correction process around the second type of defective pixel to be smaller than the weighting factor K1 of the correction process around the first type of defective pixel. The weighting factors may be preset. The correction processing unit 304 generates a second image by performing a convolution operation on the pixels around the first type of defective pixel in the first image and the pixels around the second type of defective pixel using the array data obtained by multiplying the second array data by different weighting factors (see FIGS. 9(c) and 9(d)). As will be described in detail later, the first type of defective pixel is a defective pixel that is more likely to be miscounted due to light emission crosstalk than the second type of defective pixel.

[0052] Then, in step S404, correction processing unit 304 subtracts the second image from the first image to generate a third image (see FIG. 9(e)). As described above, since the second image is a signal representing miscounting caused by light-emission crosstalk, the third image is an image obtained by restoring the signal that would have been obtained if miscounting due to light-emission crosstalk had not occurred. In other words, the processing in steps S403 and S404 reduces the impact of miscounting caused by light-emission crosstalk. Note that the correction process path from the first image of the first type of defective pixel at the bottom of FIG. 9(a) to the third image of the first type of defective pixel at the bottom of FIG. 9(e) is an example of a first correction process. The correction process path from the first image of the second type of defective pixel at the top of FIG. 9(a) to the third image of the second type of defective pixel at the top of FIG. 9(e) is an example of a second correction process.

[0053] In step S404, instead of performing the convolution operation, the first image and the second array data may be subjected to Fourier transform and then the product may be obtained.

[0054] <Crosstalk matrix> The first array data may be one-dimensional or two-dimensional as long as it contains two or more pieces of data (occurrence probability). However, as shown in FIG. 7, the first array data will be described as two-dimensional array data. For example, taking into consideration the symmetry of the occurrence probability of light-emitting crosstalk, the first array data is a matrix with an odd number of rows and columns, and is symmetrical vertically and horizontally around the center. Since the occurrence probability of light-emitting crosstalk increases closer to the pixel, the crosstalk matrix of the first array data has a maximum value (peak value) at the center in the row and column directions and a distribution that monotonically changes (e.g., monotonically decreases) toward the periphery. Furthermore, one row or one column of the crosstalk matrix, which is two-dimensional array data, is extracted as one-dimensional data. The occurrence probability of light-emitting crosstalk in this one-dimensional data in the row and column directions has a distribution that monotonically decreases from the center, which is the peak value, toward both ends. Furthermore, other one-dimensional data that share the peak value of the extracted one-dimensional data and are arranged in a direction intersecting the one-dimensional data also have a distribution in which the occurrence probability of light-emitting crosstalk monotonically decreases from the peak value toward the data ends.

[0055] <Effects of this embodiment> Next, the effects of this embodiment will be described in comparison with a comparative example in which correction processing is performed using the same weighting coefficients on the periphery of a first type of defective pixel and the periphery of a second type of defective pixel.

[0056] As described above, light-emission crosstalk is crosstalk caused by the avalanche light emission phenomenon, and therefore, in pixels surrounding a first type of defective pixel caused by the device characteristics of the APD, there is a high possibility that a miscount will occur due to light-emission crosstalk from the first type of defective pixel. On the other hand, in pixels surrounding a second type of defective pixel in which a miscount will occur due to the counter circuit 211, there is a low possibility that a miscount will occur due to light-emission crosstalk from the second type of defective pixel. In other words, the first type of defective pixel is a defective pixel that is more likely to cause a miscount due to light-emission crosstalk than the second type of defective pixel. In other words, the pixels surrounding the first type of defective pixel and the pixels surrounding the second type of defective pixel have different causes (causes) of miscounting.

[0057] FIG. 10 shows the luminance distribution around a defective pixel before signal processing (before correction processing). FIGS. 10(a) and 10(b) show the luminance distribution of pixels around a first type of defective pixel. FIGS. 10(c) and 10(d) show the luminance distribution of pixels around a second type of defective pixel. FIGS. 10(a) and 10(c) show the pixel luminance values ​​represented by different shades of color (differences in hatching). FIGS. 10(b) and 10(d) show the relative output of the luminance distribution on the X-axis of FIGS. 10(a) and 10(c).

[0058] FIG. 11 is a diagram illustrating a second image during signal processing (correction processing) in a comparative example. FIG. 12 is a diagram illustrating a third image as a result of signal processing (correction processing) in a comparative example. FIGS. 11(a), 11(c), 12(a), and 12(c) are diagrams in which pixel luminance values ​​are represented by shades of color (differences in hatching). FIGS. 11(b), 11(d), 12(b), and 12(d) are diagrams showing the relative output of the luminance distribution on the X axis in FIGS. 11(a), 11(c), 12(a), and 12(c).

[0059] 11(a) and 11(b) are diagrams illustrating a second image, which is the result of performing a convolution operation on the first type of defective pixel by multiplying the second array data by a weighting coefficient of 1. As can be seen from the diagram, the second image shows the distribution of miscounts due to light emission crosstalk caused by the first type of defective pixel. Therefore, by subtracting the second image from the first image for pixels surrounding the first type of defective pixel, a third image can be generated, as shown in FIGS. 12(a) and 12(b), in which cluster defects caused by the first type of defective pixel are reduced.

[0060] On the other hand, if a convolution operation is performed on the second type of defective pixels with a weighting factor of 1, as with the first type of defective pixels, the image quality will actually deteriorate. Figures 11(c) and 11(d) are diagrams of a second image that is the result of multiplying the second array data by a weighting factor of 1 and performing a convolution operation on the second type of defective pixels. As can be seen from the diagram, the second image shows the distribution of miscounts due to light-emission crosstalk in the case where the second type of defective pixels are pixels (first pixels) that cause miscounts due to light-emission crosstalk.

[0061] However, because the second type of defective pixel is a defective pixel caused by the counter circuit 211, no miscounting due to light-emission crosstalk actually occurs. As a result, subtracting the second image from the first image causes dark spots to appear around the second type of defective pixel, as shown in the third image in Figures 12(c) and 12(d). In other words, image quality deteriorates due to erroneous correction using the same weighting coefficient as that used for the first type of defective pixel.

[0062] In the photoelectric conversion device 300 of this embodiment, the weighting coefficient K1 of the convolution calculation is increased for pixels surrounding a first type of defective pixel that causes erroneous counting in surrounding pixels due to light emission crosstalk, thereby reducing the occurrence of cluster defects due to erroneous counting. Meanwhile, the weighting coefficient K2 of the convolution calculation is decreased for pixels surrounding a second type of defective pixel that does not cause erroneous counting in surrounding pixels due to light emission crosstalk, thereby reducing the occurrence of black spots due to erroneous correction. In this way, this embodiment can suppress degradation of image quality due to correction of pixels surrounding the first type of defective pixel and the second type of defective pixel by changing the weighting coefficients of the correction process between the pixels surrounding the first type of defective pixel and the pixels surrounding the second type of defective pixel.

[0063] <Pixels other than defective pixels> As described above, luminescence crosstalk is caused by the avalanche phenomenon, and therefore luminescence crosstalk also occurs due to the avalanche phenomenon that occurs when photons are incident on a pixel. Therefore, the signal processing of FIG. 8 may be performed on pixels surrounding normal pixels other than the pixels surrounding the first type defective pixel and the pixels surrounding the second type defective pixel.

[0064] It is preferable to set the weighting coefficients for the correction process around normal pixels to be larger than the weighting coefficients for the correction process around the first type of defective pixel. That is, the weighting coefficients may be assigned to normal pixels, first type of defective pixels, and second type of defective pixels in descending order. The reason for this is explained below.

[0065] As mentioned above, the probability of light-emission crosstalk occurring can be predicted based on the pixel structure of the photoelectric conversion element. However, due to factors such as manufacturing variations, the probability of light-emission crosstalk occurring is not necessarily the same for all pixels.

[0066] In the case of a pixel whose output level differs little from that of surrounding pixels, the influence of that pixel on the surrounding pixels during convolution calculation is about the same as the influence of the surrounding pixels on that pixel, so the influence of manufacturing variations is limited.

[0067] On the other hand, pixels with a higher output level than surrounding pixels, i.e., the first type of defective pixel that may cause a cluster defect, have a large effect on surrounding pixels during convolution operations, and are therefore significantly affected by manufacturing variations.

[0068] That is, in pixels where the probability of occurrence of light-emitting crosstalk is low, the probability of light-emitting crosstalk used during the convolution calculation is higher than the light-emitting crosstalk that actually occurs, so black sunken pixels occur in the pixels surrounding the cluster defect. On the other hand, in pixels where the probability of occurrence of light-emitting crosstalk is high, the probability of light-emitting crosstalk used during the convolution calculation is lower than the light-emitting crosstalk that actually occurs, so residual defects occur in the pixels surrounding the cluster defect. In particular, when black sunken pixels occur, only the area around the defective pixel is sunken, which creates an unnatural feeling in the human visual sense.

[0069] Therefore, it is better to reduce the weighting coefficients around the first type of defective pixel compared to the normal pixels to suppress darkening while further enhancing the effect of suppressing emission crosstalk in the normal pixels. Specifically, the weighting coefficient K1 around the first type of defective pixel may be set to approximately 0.5 to 0.9 times the weighting coefficient K3 for the normal pixels.

[0070] Since the only pixels that are significantly affected by manufacturing variations are those with a higher output level than the surrounding pixels, i.e., the first type of defective pixels that may cause cluster defects, weighting coefficient K1 should be set smaller than weighting coefficient K3 for only the first type of defective pixels.Furthermore, since erroneous counting caused by counter circuit 211 occurs only in the second type of defective pixels, weighting coefficient K2 for the second type of defective pixels should be set smaller than weighting coefficient K1.

[0071] That is, the pixels surrounding the first type of defective pixel and the pixels surrounding the second type of defective pixel may be only one pixel, such as the first type of defective pixel or the second type of defective pixel itself.

[0072] <Weighting coefficient K2> The weighting coefficient K2 for the second type of defective pixel may be changed depending on the ratio of the output value of the second type of defective pixel that is generated by photons incident on the second pixel to the output value that is generated by miscounting caused by the counter circuit 211. When a large number of photons enters the second type of defective pixel, the ratio of the output value of the second type of defective pixel that is generated by miscounting caused by the counter circuit 211 is small, so it is better to set the weighting coefficient K2 closer to the weighting coefficient K1.

[0073] On the other hand, when the number of photons entering the second type of defective pixel is small, most of the output values ​​of the second type of defective pixel are generated due to miscounting caused by the counter circuit 211, so a smaller weighting coefficient K2 is preferable. That is, the larger the difference between the output value of the second type of defective pixel and the output values ​​of the surrounding pixels of the second type of defective pixel, the smaller the weighting coefficient K2 can be. In particular, when the output value of the second type of defective pixel due to miscounting caused by the counter circuit 211 is greater than the predetermined output threshold, regardless of the output values ​​of the surrounding pixels of the second type of defective pixel, it is better to set the weighting coefficient K2 to 0, i.e., not to perform the second correction process.

[0074] <How to change the weighting coefficient of the convolution operation> In step S403, to change the weighting coefficients for the convolution operation, the weighting coefficients for the second array data may be changed for each pixel, as shown in Figure 9. Alternatively, a fourth image may be generated by multiplying the output value of each pixel of the first image by a weighting coefficient for the convolution operation, and the fourth image may be convolved with the second array data. In this case, the processing load can be reduced.

[0075] Fig. 13 is a diagram showing an example of image changes in another first correction process of the first embodiment. Fig. 13 is a diagram showing changes in an image in which a weighting coefficient K1 is applied to the periphery of a first type of defective pixel. Fig. 14 is a diagram showing changes in an image in another second correction process of the first embodiment. Fig. 14 is a diagram showing changes in an image in which a weighting coefficient K2 is applied to the periphery of a second type of defective pixel.

[0076] Figures 13(a) and 14(a) are diagrams of the first image. Figures 13(b) and 14(b) are diagrams of the fourth image. Figures 13(c) and 14(c) are diagrams of the second array data. Figures 13(d) and 14(d) are diagrams of the second image. Figures 13(e) and 14(e) are diagrams of the third image.

[0077] As shown in FIGS. 13 and 14, the correction processing unit 304 may multiply the first image by a weighting coefficient, and then perform a convolution operation with the second array data to generate a second image.

[0078] Specifically, as shown in FIG. 13 , in the first correction process, the correction processing unit 304 generates a fourth image by multiplying the first type of defective pixel and pixels surrounding the defective pixel in the first image by a first weighting coefficient K1. The correction processing unit 304 generates a second image by performing a convolution operation on the fourth image using the second array data. The correction processing unit 304 generates a third image by subtracting the second image from the first image. FIG. 13 illustrates an example in which the pixels surrounding the first type of defective pixel are the first type of defective pixel itself. That is, in FIG. 13 , in the first correction process, the fourth image is generated by multiplying the first type of defective pixel in the first image by a first weighting coefficient K1.

[0079] As shown in FIG. 14 , in the second correction process, the correction processing unit 304 generates a fourth image by multiplying the second type of defective pixel and the pixels surrounding the defective pixel in the first image by the second weighting coefficient K2. The correction processing unit 304 generates a second image by performing a convolution operation on the fourth image using the second array data. The correction processing unit 304 generates a third image by subtracting the second image from the first image. FIG. 14 illustrates an example in which the pixels surrounding the second type of defective pixel are the second type of defective pixel itself. That is, in FIG. 13 , in the second correction process, the fourth image is generated by multiplying the second type of defective pixel in the first image by the second weighting coefficient K2.

[0080] <Defect extraction patterns> While Fig. 8 shows an example in which address data is used to extract first and second types of defective pixels in step S404, correction processing unit 304 may instead extract defective pixels from the first image. This is explained below. Fig. 15 is a flowchart of the defective pixel extraction process. Fig. 16 is a diagram of the area around a pixel of interest, illustrating the details of the defective pixel extraction process.

[0081] First, in step S501, the correction processing unit 304 performs a first defect extraction process in which a pixel (also referred to as a pixel of interest) having a higher output value than surrounding pixels is extracted, thereby extracting a first type of defective pixel and a second type of defective pixel together as defective pixels. Specifically, the correction processing unit 304 generates a fifth image by applying a median filter to the first image. Then, the correction processing unit 304 extracts, as defective pixels, pixels whose output difference between the first image and the fifth image is equal to or greater than a first difference threshold.

[0082] 16(a) is a diagram illustrating a first defect extraction process for extracting defective pixels. As another extraction method, as shown in FIG. 16(a), the correction processing unit 304 may calculate the average value of the output values ​​of pixels (also called nearest pixels) surrounding a pixel of interest, and extract as defective pixels pixels whose difference between this average value and the output value of the pixel of interest is equal to or greater than a first difference threshold. The correction processing unit 304 may also calculate a median (center value) instead of the average value of the output values ​​of nearest pixels surrounding the pixel of interest, and extract as defective pixels pixels whose difference between the median and the output value of the pixel of interest is equal to or greater than the first difference threshold. The nearest pixels referred to here are pixels adjacent to the central pixel of interest on the left, right, top, and bottom.

[0083] Next, in step S502, the correction processing unit 304 performs a second defect extraction process to distinguish between first-type defective pixels and second-type defective pixels. As described above, the first-type defective pixels cause miscounting due to light-emission crosstalk, whereas the second-type defective pixels do not cause miscounting due to light-emission crosstalk. Therefore, the correction processing unit 304 can distinguish between defective pixels by determining whether the defective pixels are cluster defects.

[0084] FIG. 16(b) illustrates a second defect extraction process for distinguishing defective pixels. The nearest neighboring pixels in FIG. 16(b) are pixels adjacent to the central pixel of interest above, below, to the left, and to the right. The second neighboring pixels are pixels adjacent to the central pixel of interest in the diagonal direction and pixels adjacent to the nearest neighboring pixel on the opposite side of the pixel of interest. In other words, the second neighboring pixels are the pixels next closest to the pixel of interest after the nearest neighboring pixel. As shown in FIG. 16(b), the correction processing unit 304 determines a pixel for which the difference between the average output value of the nearest neighboring pixel of the pixel of interest and the average output value of the second neighboring pixel is equal to or greater than a second difference threshold as a first type defective pixel. The correction processing unit 304 determines a defective pixel for which the difference between the average output value of the nearest neighboring pixel of the pixel of interest and the average output value of the second neighboring pixel is smaller than the second difference threshold as a second type defective pixel. This is because, as shown in FIG. 10, in the case of a first type defective pixel, a difference in output level occurs between the nearest neighboring pixel and the second neighboring pixel of the defective pixel due to light-emission crosstalk. On the other hand, in the case of a second type of defective pixel, there is no difference in output level due to light emission crosstalk between the nearest pixel and the second nearest pixel of the defective pixel. In other words, the second defect extraction process is a process that distinguishes between the first type of defective pixel and the second type of defective pixel by comparing the output values ​​or average output values ​​between pixels surrounding the pixel of interest.

[0085] If the defective pixel is a first type defective pixel, the difference in output level between the defective pixel's nearest neighbor pixel and its second nearest neighbor pixel is determined by the product of the probability of emission crosstalk occurrence and the output value of the first type defective pixel. Therefore, the second difference threshold may be determined by both the first array data and the output value of the defective pixel extracted in step S501.

[0086] Extracting defective pixels and their output levels from the first image does not require memory for storing address data for defective pixels, compared to using address data, and therefore simplifies the configuration of photoelectric conversion device 300. On the other hand, when extracting defective pixels from the first image without using address data, there is a possibility that defective pixels will be erroneously detected if a subject such as a bright spot is photographed, so it is better to use address data in order to accurately extract defective pixels.

[0087] <Correcting defective pixels by interpolation> As described above, the signal processing in the photoelectric conversion device of this embodiment is signal processing that suppresses erroneous counting due to light-emission crosstalk. Therefore, as shown in FIGS. 9, 13, and 14, the first type of defective pixel and the second type of defective pixel themselves remain as defective pixels even after signal processing. Therefore, after step S404 in FIG. 8, the correction processing unit 304 may perform defect interpolation processing to correct the first type of defective pixel and the second type of defective pixel. As the defect interpolation processing, the correction processing unit 304 may replace the output value of the defective pixel with the average or median of the output values ​​of the surrounding pixels.

[0088] <Color sensor> The photoelectric conversion element 100 may be a monochrome sensor without an on-chip color filter, or may be a so-called color sensor having at least two or more types of pixels with different spectral characteristics. In the case of a color sensor, the correction processing unit 304 may change the correction processing for each pixel with different spectral characteristics. Specifically, the correction processing unit 304 may change either the size of the array data used in the convolution calculation or the weighting coefficient for each pixel with different spectral characteristics.

[0089] Furthermore, when extracting defective pixels from the first image in step S404, the correction processing unit 304 may extract a first type of defective pixel and a second type of defective pixel for each pixel with different spectral characteristics. For example, in the case of a so-called Bayer array color sensor, in step S501, as shown in FIG. 16(c), the correction processing unit 304 may extract defective pixels (here, including first type of defective pixels and second type of defective pixels) by comparing the pixel of interest with its nearest neighbor pixel that has the same spectral characteristics (same color) as the pixel of interest.

[0090] In step S502, as shown in FIG. 16(d), the correction processing unit 304 may distinguish between the first type of defective pixel and the second type of defective pixel by comparing a nearest neighboring pixel, which includes a pixel having spectral characteristics different from those of the pixel of interest determined to be the defective pixel, with a second nearest neighboring pixel having the same spectral characteristics as the nearest neighboring pixel.

[0091] 16(d), the correction processing unit 304 determines the pixel of interest to be a first type defective pixel if the maximum value of the differences between the pixel of interest and the average value of second adjacent pixels having the same spectral characteristics as the nearest adjacent pixel is equal to or greater than a second difference threshold, and otherwise determines the pixel of interest to be a second type defective pixel. Note that the correction processing unit 304 may determine the pixel of interest to be a first type defective pixel if the average value of the differences between the pixel of interest and second adjacent pixels having the same spectral characteristics as the nearest adjacent pixel is equal to or greater than the second difference threshold, and may determine other pixels of interest to be second type defective pixels. Alternatively, the correction processing unit 304 may also use the value of a third adjacent pixel adjacent to the second adjacent pixel to distinguish between a pixel of the first type and a pixel of the second type defective.

[0092] As described above, in the first embodiment, different correction processes are performed on pixels surrounding a first type of defective pixel and pixels surrounding a second type of defective pixel, which have different causes. This allows the first embodiment to suppress degradation of image quality while correcting pixels surrounding a defective pixel in a photoelectric conversion device 300 that includes an APD 201.

[0093] (Second embodiment) <Clock-driven type> The photoelectric conversion device according to the second embodiment differs from the photoelectric conversion device according to the first embodiment in the method of driving the photoelectric conversion element. FIG. 17 is a diagram showing an equivalent circuit of the photoelectric conversion unit and signal processing circuit of a pixel according to the second embodiment. Specifically, as shown in FIG. 17, in the photoelectric conversion device according to the second embodiment, the quench element 202 is configured with a MOS transistor. The on / off of the quench element 202 is controlled by a control signal CLK input to the gate of the quench element 202. The control signal CLK is controlled by a signal generator 1701 in the control pulse generator 115. The signal generator 1701 is connected to the gate of the quench element 202.

[0094] <Clock Drive Explanation> 18 is a timing chart of the photoelectric conversion element of the second embodiment. Specifically, FIG. 18 is a timing chart that schematically illustrates the relationship between the control signal CLK input to the quench element 202, the voltage at node nodeA, the voltage at node nodeB, and the output signal. In the photoelectric conversion element of the second embodiment, when the control signal CLK is at a high level, the drive voltage VH is unlikely to be supplied to the APD 201. On the other hand, when the control signal CLK is at a low level, the drive voltage VH is supplied to the APD 201. The high level of the control signal CLK is, for example, 1 V, and the low level of the control signal CLK is, for example, 0 V.

[0095] When the control signal CLK is at a high level, the quench element 202 as a switch is turned off, and when the control signal CLK is at a low level, the quench element 202 is turned on. The resistance value of the quench element 202 when the control signal CLK is at a high level is higher than the resistance value of the quench element 202 when the control signal CLK is at a low level. When the control signal CLK is at a high level, a recharge operation is unlikely to occur even if avalanche multiplication occurs in the APD 201, so the voltage supplied to the APD 201 is a voltage equal to or lower than the breakdown voltage of the APD 201. Therefore, the avalanche multiplication operation in the APD 201 stops.

[0096] At time t1, the control signal CLK changes from high to low, turning on the quench element 202 and initiating the recharge operation of the APD 201. This causes the voltage at the cathode of the APD 201 to transition to high. The cathode voltage is the same as the voltage at node A. The voltage difference between the voltages applied to the anode and cathode of the APD 201 allows avalanche multiplication. That is, when the cathode voltage transitions from low to high, the voltage at node A exceeds the decision threshold at time t2. At this time, the pulse signal output from node B is inverted, transitioning from high to low. When recharging is complete, the voltage difference (drive voltage VH - drive voltage VL) is applied to the APD 201. After that, the control signal CLK transitions to high, turning off the quench element 202.

[0097] Next, at time t3, when a photon is incident on the APD 201, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage at the cathode drops. That is, the voltage at node A drops. When the voltage at node A drops below the decision threshold, the voltage at node B changes from low to high. That is, the portion of the output waveform at node A that exceeds the decision threshold is waveform-shaped by the waveform shaping unit 210 and output as a signal from node B. The output signal is then counted by the counter circuit 211. That is, the count value of the counter signal output from the counter circuit 211 increases by 1 LSB (Least Significant Bit).

[0098] Although photons are incident on the APD 201 between time t3 and time t4, the quench element 202 is in the off state and the voltage difference applied to the APD 201 is not large enough to cause avalanche multiplication, so the voltage level of the node nodeA does not exceed the decision threshold.

[0099] At time t4, the control signal CLK changes from high to low, turning on the quench element 202. As a result, a current flows through node A to compensate for the voltage drop from the drive voltage VL, and the voltage at node A transitions to high. At this time, at time t5, the voltage at node A becomes equal to or greater than the determination threshold, so the pulse signal at node B is inverted and transitions from high to low.

[0100] At time t6, node A settles at a high level, and the control signal CLK changes from a low level to a high level. After this, the voltages of the nodes and signal lines change in response to the control signal CLK and the incidence of photons, as described from time t1 to time t6.

[0101] As described above, in the second embodiment, the recharge frequency of the APD 201 can be controlled by applying the control signal CLK to the quench element 202 to switch the quench element 202 on and off. If the control signal CLK is not used, there is a problem that the actual count value becomes smaller than the count value corresponding to the luminance of incident light at high luminance. However, in the second embodiment, this problem can be solved by applying the control signal CLK to the quench element 202 to switch the quench element 202 on and off.

[0102] However, when the recharge frequency of the APD 201 is controlled by the control signal CLK, the relationship between the number of input signals and the number of output signals is not linear. If the effects of light-emission crosstalk are ignored, the relationship between the number of input signals and the number of output signals can be theoretically derived. Specifically, when the number of input signals is Nph, the number of output signals is Nct, the frequency of the control signal CLK is f (the inverse of the number of CLKs per unit time), and the length of the exposure period is T, the number of output signals Nct is expressed by the following equation (1):

number

[0103] The photoelectric conversion device shown in the second embodiment performs correction processing that can simultaneously reduce the influence of nonlinear response caused by the control signal CLK and the influence of erroneous counting caused by crosstalk, as will be explained below.

[0104] <Flowchart> 19 is a flowchart of signal processing in the photoelectric conversion device of the second embodiment. The following description will focus on steps that are different from the flowchart of FIG.

[0105] First, in step S601, the image generating unit 303 generates a first image, as in FIG.

[0106] In step S602, the correction processing unit 304 extracts first-type defective pixels and second-type defective pixels.

[0107] Next, in step S603, similar to step S403, the first image is subjected to correction processing using the first array data to generate a second image. In this step, similar to the first embodiment, the correction processing unit 304 sets the weighting coefficients for the correction processing of the second type of defective pixel to be smaller than the weighting coefficients for the correction processing of the first type of defective pixel.

[0108] In the photoelectric conversion device of the second embodiment, the correction processing unit 304 branches to steps S602 and S603, and then performs nonlinearity correction processing in step S604. In step S604, the correction processing unit 304 performs nonlinearity correction processing on the first image to restore the nonlinear response generated by the control signal CLK to a linear state, thereby generating a sixth image. Specifically, the correction processing unit 304 can calculate the number of input signals Nph from the number of output signals Nct using the following equation (2):

number

[0109] Then, in step S605, the correction processing unit 304 subtracts the second image from the sixth image to generate a third image. As described above, the second image is a signal representing a counting error caused by light-emission crosstalk, and therefore the third image is an image obtained by restoring a signal that would be obtained if no counting error caused by light-emission crosstalk had occurred. That is, by performing the processes of steps S603 to S605, the second embodiment can simultaneously reduce the influence of the nonlinear response caused by the control signal CLK and the influence of the counting error caused by crosstalk.

[0110] (Third embodiment) <Photoelectric conversion system> 20 is a block diagram of the overall configuration of a photoelectric conversion system according to the third embodiment. The photoelectric conversion system according to the third embodiment has the photoelectric conversion device according to the first or second embodiment. The photoelectric conversion system 800 has a photoelectric conversion device 300 having a photoelectric conversion element 100, a control unit 801, a storage unit 802, and a communication unit 803.

[0111] The photoelectric conversion element 100 captures an optical image formed by the imaging optical system 301 and outputs an electrical signal.

[0112] The signal processing unit 302 generates an image from the signal output from the photoelectric conversion element 100, and performs correction processing on the image as well as black level correction, gamma curve adjustment, noise reduction, data compression, etc. to generate a final image. If the photoelectric conversion element 100 has an on-chip RGB color filter, the signal processing unit 302 may perform processing such as white balance correction and color conversion.

[0113] The control unit 801 functions as a computer and controls the operation of each unit of the photoelectric conversion system 800 based on a computer program. The control unit 801 also controls the length of the exposure period of each frame of the photoelectric conversion element 100 and the timing of the control signal CLK via a control pulse generation unit of the photoelectric conversion element 100. The specific hardware configuration of the control unit 801 will be described later.

[0114] The storage unit 802 includes a recording medium such as a memory card, a hard disk, etc. The communication unit 803 has a wireless or wired interface, and outputs the generated image to the outside of the photoelectric conversion system 800 and receives signals from the outside.

[0115] 21 is a block diagram showing the hardware configuration of the control unit 801. The control unit 801 is an example of a computer. The control unit 801 has a processor 2101, a memory 2102, a storage 2103, a communication IF 2104, an input IF 2105, an output IF 2106, and a bus 2107. The processor 2101, the memory 2102, the storage 2103, the communication IF 2104, the input IF 2105, and the output IF 2106 are connected via the bus 2107 so as to be able to transmit and receive information to and from each other.

[0116] The processor 2101 is an arithmetic processing device, such as a CPU (Central Processing Unit). The control unit 801 may include other processors, such as an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), or a QPU (Quantum Processing Unit), instead of or in addition to the CPU. The processor 2101 reads out programs stored in the storage 2103 and loads them into the memory 2102 to implement various functions. For example, the processor 2101 may implement some or all of the functions of the image generation unit 303 and the correction processing unit 304 of the signal processing unit 302 by reading out the programs. Some or all of the functions of the image generation unit 303 and the correction processing unit 304 of the signal processing unit 302 may be implemented by one or more circuits, such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array).

[0117] The memory 2102 is a high-speed readable / writable storage device such as a RAM (Random Access Memory). The memory 2102 functions as a work area when the processor 2101 executes a program. The memory 2102 temporarily stores the program and parameters necessary for executing the program.

[0118] The storage 2103 is a non-volatile storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage 2103 retains programs, parameters required for executing the programs, and results of executing the programs even when power is not supplied. The storage 2103 stores, for example, image data, first array data, weighting coefficients, and the like output by the signal processing unit 302. The storage 2103 may also function as the memory unit 802.

[0119] The communication IF 2104 is an interface for realizing communication with an external device via a wired or wireless network. The communication IF 2104 may also function as the communication unit 803.

[0120] The input IF 2105 is an interface for receiving information input from an input device, such as a mouse, a keyboard, or a touch panel.

[0121] The output IF 2106 is an interface for outputting information to an external device, such as an image display device.

[0122] Note that a computer program that realizes part or all of the control in this embodiment and the functions of the above-described embodiment may be supplied to the photoelectric conversion device via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the photoelectric conversion device may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention.

[0123] (Other Examples) In the above embodiment, an example was given in which the correction processing unit generates a plurality of different second array data from the first array data using a plurality of different weighting coefficients, and then performs the correction processing, but the method of the correction processing is not limited to this. For example, the correction processing unit may perform the correction processing based on a plurality of second array data that are set in advance.

[0124] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0125] The disclosure of this specification includes the following processing device, photoelectric conversion device, processing method, and program. (Item 1) A processing device that generates and corrects an image based on a count value output by a photoelectric conversion element having an avalanche photodiode that converts light from a subject into an electrical signal, counts the signal, and outputs the count value, an image generating means for generating a first image based on the count value; a correction processing means for executing a correction process to correct the first image; and The correction processing means a first correction process for first pixels including pixels surrounding a first type of defective pixel that causes a miscount due to a first cause; a second correction process different from the first correction process for second pixels including pixels surrounding a second type of defective pixel that causes a miscount due to a second cause different from the first cause; Perform the correction process including A processing device characterized by: (Item 2) The correction processing means executes the first correction processing using a first weighting coefficient, and executes the second correction processing using a second weighting coefficient different from the first weighting coefficient. 2. The processing device according to item 1, (Item 3) The correction processing means executes the first correction processing and the second correction processing based on first array data of a two-dimensional array that associates the positions of the first pixel and the second pixel with correction values ​​for correcting the pixels. 3. The processing device according to item 1 or 2, (Item 4) The center of at least one of one row and one column of one-dimensional data of the first array data has a maximum value. 4. The processing device according to item 3, (Item 5) The value of the one-dimensional data of the first array data decreases from the center toward the edge. 5. The processing device according to item 4. (Item 6) The center of the one-dimensional data in the row direction and the center of the one-dimensional data in the column direction of the first array data share a maximum value, and the value decreases toward the end. 4. The processing device according to item 3, (Item 7) The first array data has an odd number of rows and an odd number of columns, and is symmetrical about the center in the vertical and horizontal directions. 7. The processing device according to any one of items 3 to 6, (Item 8) In the correction process, the correction processing means generates a second image by performing a convolution operation on the first image using array data obtained by multiplying second array data generated from the first array data by a weighting coefficient. 8. The processing device according to any one of items 3 to 7, (Item 9) The correction processing means generates the second array data by setting the central value of the first array data to 0. 9. The processing device according to item 8, (Item 10) The correction processing means multiplies the second array data by a first weighting coefficient in the first correction processing, and multiplies the second array data by a second weighting coefficient different from the first weighting coefficient in the second correction processing. 10. The processing device according to item 8 or 9, (Item 11) The correction processing means the first correction processing generates a third image by subtracting the first pixel of the second image from the first pixel of the first image; In the second correction process, a third image is generated by subtracting the second pixel of the second image from the second pixel of the first image. 11. The processing device according to item 10. (Item 12) In the correction process, the correction processing means generates a second image by performing a convolution operation on a fourth image generated by multiplying pixels of the first image by weighting coefficients, using second array data generated from the first array data. 12. The processing device according to any one of items 3 to 11, (Item 13) The first pixel is the first type of defective pixel itself. 13. The processing device according to any one of items 1 to 12, (Item 14) The second pixel is the second type of defective pixel itself. 2. The processing device according to item 1, (Item 15) The larger the difference between the output value of the second type of defective pixel and the output value of the second pixel other than the second type of defective pixel, the smaller the second weighting coefficient in the second correction process. 3. The processing device according to item 2. (Item 16) The correction processing means does not execute the second correction processing when the output value of the second type of defective pixel is greater than a predetermined output threshold value. 16. The processing device according to any one of items 1 to 15, (Item 17) The first array data includes the correction value based on the probability of occurrence of miscounting due to light emission crosstalk. 4. The processing device according to item 3, (Item 18) The first type of defective pixel has a higher probability of occurrence of erroneous counting due to light emission crosstalk than the second type of defective pixel. 18. The processing device according to any one of items 1 to 17, (Item 19) The second weighting factor is smaller than the first weighting factor. 3. The processing device according to item 2. (Item 20) The correction processing means extracts the first type of defective pixel and the second type of defective pixel based on address data acquired in advance. 20. The processing device according to any one of items 1 to 19, (Item 21) The correction processing means a first defect extraction process for extracting defective pixels by comparing an output value of a pixel with output values ​​of pixels surrounding the pixel using the first image; a second defect extraction process for comparing output values ​​between a plurality of pixels surrounding the defective pixel to distinguish the defective pixel into the first type and the second type; 21. The processing device according to any one of items 1 to 20, wherein the processing device executes the above. (Item 22) The correction processing means corrects the output values ​​of the first type defective pixel and the second type defective pixel based on the output values ​​of surrounding pixels. 22. The processing device according to any one of items 1 to 21, (Item 23) The correction processing means performs a third correction processing different from the first correction processing and the second correction processing on normal pixels other than the first type defective pixels and the second type defective pixels. 4. The processing device according to item 3, (Item 24) The correction processing means executes the third correction processing using a third weighting coefficient that is larger than the first weighting coefficient used in the first correction processing. 24. The processing device according to item 23, (Item 25) The pixels include two or more types of pixels with different spectral characteristics. 25. The processing device according to any one of items 1 to 24, (Item 26) 26. The processing device according to item 25, wherein the correction processing means executes the correction processing using a different weighting coefficient for each pixel having a different spectral characteristic. (Item 27) The correction processing means a first defect extraction process for extracting defective pixels by comparing output values ​​of pixels having the same spectral characteristics with output values ​​of pixels surrounding the defective pixels using the first image; a second defect extraction process for comparing output values ​​between pixels having different spectral characteristics among a plurality of pixels surrounding the defective pixel, and distinguishing the defective pixel into the first type and the second type; 27. The processing device according to any one of items 1 to 26, wherein the processing device executes the above. (Item 28) The correction processing means performs nonlinearity correction processing on the first image, which is a correction of nonlinearity, based on the following equation including the number of input signals Nph, the number of output signals Nct, the frequency f of the control signal CLK, and the length T of the exposure period:

number

[0126] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0127] 100 Photoelectric conversion element, 101 Pixel, 102 Photoelectric conversion section, 201 APD, 300 Photoelectric conversion device, 301 Imaging optical system, 302 Signal processing section, 303 Image generation section, 304 Correction processing section, 800 Photoelectric conversion system, 801 Control section.

Claims

1. A processing device that generates and corrects an image based on a count value output by a photoelectric conversion element having an avalanche photodiode that converts light from a subject into an electrical signal, counts the signal, and outputs the count value, an image generating means for generating a first image based on the count value; a correction processing means for executing a correction process to correct the first image; and The correction processing means a first correction process for first pixels including pixels surrounding a first type of defective pixel that causes a miscount due to a first cause; a second correction process different from the first correction process for second pixels including pixels surrounding a second type of defective pixel that causes a miscount due to a second cause different from the first cause; Perform the correction process including A processing device characterized by:

2. The correction processing means executes the first correction processing using a first weighting coefficient, and executes the second correction processing using a second weighting coefficient different from the first weighting coefficient.

2. The processing device according to claim 1.

3. The correction processing means executes the first correction processing and the second correction processing based on first array data, which is a two-dimensional array that associates the positions of the first pixel and the second pixel with correction values ​​for correcting the pixels.

2. The processing device according to claim 1.

4. The center of at least one of one row and one column of one-dimensional data of the first array data has a maximum value.

4. The processing device according to claim 3.

5. The value of the one-dimensional data of the first array data decreases from the center toward the edge.

5. The processing device according to claim 4.

6. The center of the one-dimensional data in the row direction of the first array data and the center of the one-dimensional data in the column direction share a maximum value, and the value decreases toward the end.

4. The processing device according to claim 3.

7. The first array data has an odd number of rows and columns and is symmetrical about the center in the vertical and horizontal directions.

4. The processing device according to claim 3.

8. In the correction process, the correction processing means generates a second image by performing a convolution operation on the first image using array data obtained by multiplying second array data generated from the first array data by a weighting coefficient.

4. The processing device according to claim 3.

9. The correction processing means generates the second array data by setting the central value of the first array data to 0.

9. The processing device according to claim 8.

10. The correction processing means multiplies the second array data by a first weighting coefficient in the first correction processing, and multiplies the second array data by a second weighting coefficient different from the first weighting coefficient in the second correction processing.

9. The processing device according to claim 8.

11. The correction processing means the first correction process generates a third image by subtracting the first pixel of the second image from the first pixel of the first image; In the second correction process, a third image is generated by subtracting the second pixel of the second image from the second pixel of the first image. The processing device according to claim 10 .

12. In the correction process, the correction processing means generates a second image by performing a convolution operation on a fourth image generated by multiplying pixels of the first image by weighting coefficients, using second array data generated from the first array data.

4. The processing device according to claim 3.

13. The first pixel is the first type of defective pixel itself.

2. The processing device according to claim 1.

14. The second pixel is the second type of defective pixel itself.

2. The processing device according to claim 1.

15. The larger the difference between the output value of the second type of defective pixel and the output value of the second pixel other than the second type of defective pixel, the smaller the second weighting coefficient in the second correction process.

3. The processing device according to claim 2.

16. The correction processing means does not execute the second correction processing when the output value of the second type defective pixel is greater than a predetermined output threshold value.

2. The processing device according to claim 1.

17. The first array data includes the correction value based on the probability of occurrence of miscounting due to light emission crosstalk.

4. The processing device according to claim 3.

18. The first type of defective pixel has a higher probability of occurrence of erroneous counting due to light emission crosstalk than the second type of defective pixel.

2. The processing device according to claim 1.

19. The second weighting factor is smaller than the first weighting factor.

3. The processing device according to claim 2.

20. The correction processing means extracts the first type of defective pixel and the second type of defective pixel based on address data acquired in advance.

2. The processing device according to claim 1.

21. The correction processing means a first defect extraction process for extracting defective pixels by comparing an output value of a pixel with output values ​​of pixels surrounding the pixel using the first image; a second defect extraction process for comparing output values ​​between a plurality of pixels surrounding the defective pixel to distinguish the defective pixel into the first type and the second type; 2. The processing device according to claim 1, wherein the processing device executes the following.

22. The correction processing means corrects the output values ​​of the first type defective pixel and the second type defective pixel based on the output values ​​of surrounding pixels.

2. The processing device according to claim 1.

23. The correction processing means performs a third correction processing different from the first correction processing and the second correction processing on normal pixels other than the first type defective pixels and the second type defective pixels.

4. The processing device according to claim 3.

24. The correction processing means executes the third correction processing using a third weighting factor that is larger than the first weighting factor used in the first correction processing.

24. The processing device of claim 23.

25. The pixels include two or more types of pixels having different spectral characteristics.

2. The processing device according to claim 1.

26. 26. The processing device according to claim 25, wherein the correction processing means executes the correction processing using a different weighting coefficient for each pixel having a different spectral characteristic.

27. The correction processing means a first defect extraction process for extracting defective pixels by comparing output values ​​of pixels having the same spectral characteristics with output values ​​of pixels surrounding the defective pixels using the first image; a second defect extraction process for comparing output values ​​between pixels having different spectral characteristics among a plurality of pixels surrounding the defective pixel to distinguish the defective pixel from the first type and the second type; 2. The processing device according to claim 1, wherein the processing device executes the following steps:

28. The correction processing means performs nonlinearity correction processing on the first image based on the following equation, which includes the number of input signals Nph, the number of output signals Nct, the frequency f of the control signal CLK, and the length T of the exposure period: [Equation 1] 2. The processing device according to claim 1.

29. The processing device according to claim 1 ; a photoelectric conversion element having an avalanche photodiode; and The photoelectric conversion element is a switch for switching a resistance value between one of two nodes, an anode and a cathode of the avalanche photodiode, and a power supply line; a signal generating means for generating a pulse signal for switching the resistance value of the switch; A photoelectric conversion device comprising:

30. A processing method for generating and correcting an image based on a count value output by a photoelectric conversion element having an avalanche photodiode that converts light from a subject into an electrical signal, counts the signal, and outputs the count value, comprising: an image generating step of generating a first image based on the count value; a correction processing step of executing a correction process to correct the first image; and In the correction processing step, a first correction process for first pixels including pixels surrounding a first type of defective pixel that causes a miscount due to a first cause; a second correction process different from the first correction process for second pixels including pixels surrounding a second type of defective pixel that causes a miscount due to a second cause different from the first cause; Perform the correction process including A processing method characterized by:

31. A program for causing a computer to function as each of the means of the processing device according to any one of claims 1 to 27.

Images