Photon counting apparatus, photon counting method, and photon counting processing program

Abstract

This photon counting device comprises: a plurality of pixels that each include a photoelectric conversion element and an amplifier amplifying a charge converted by the photoelectric conversion element and converting the result into a voltage; an A / D converter that converts the voltages outputted from the amplifiers of the plurality of pixels into digital values; a first derivation unit that, on the basis of the digital values, derives a provisional value for the number of photons for each pixel among the plurality of pixels; and a second derivation unit that, on the basis of a first probability that is based on the distribution of the number of photons and a second probability that is based on reading noise, derives a settled value for the number of photons for a target pixel.

Description

[Technical Field]

[0001] This disclosure relates to a photon counting apparatus, a photon counting method, and a photon counting processing program. [Background technology]

[0002] For example, Patent Documents 1 and 2 describe a photon counting device using a CMOS (Complementary Metal Oxide Semiconductor) image sensor. In this device, when photons are input to a photoelectric conversion element, photoelectrons generated according to the number of input photons are stored as electric charge. The electric charge stored in the photoelectric conversion element is converted into a voltage and amplified by an amplifier. The voltage output from the amplifier is converted into a digital value by an A / D converter. In the photon counting device, the number of photons in the pixels constituting the image sensor is determined based on the digital value output from the A / D converter.

[0003] Furthermore, Non-Patent Documents 1 to 3 describe techniques related to photon counting using CMOS image sensors. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] International Publication No. 2019 / 102636 [Patent Document 2] International Publication No. 2019 / 102637 [Non-patent literature]

[0005] [Non-Patent Document 1] B Saleh Masoodian, Jiaju Ma, Dakota Starkey, Yuichiro Yamashita, and Eric R. Fossum, “A 1Mjot 1040fps 0.22e-rms Stacked BSI Quanta Image Sensor with Cluster-Parallel Readout”, Proceedings of the 2017 International Image Sensor Workshop (IISW), May 30 - June 2, 2017, P230-233 [Non-Patent Document 2] JIAJU MA et al., “Photon-number-resolving megapixel image sensor at room temperature without avalanche gain”, Optica, Vol. 4, No. 12, December 2017, p1474 -p1481 [Non-Patent Document 3] DAKOTA A. STARKEY et al., “Determining Conversion Gain and Read Noise Using a Photon-Counting Histogram Method for Deep Sub-Electron Read Noise Image Sensors”, JOURNAL OF THE ELECTRON DEVICES SOCIETY, VOLUME 4, NO. 3, MAY 2016, p129 -p135 [Overview of the project] [Problems that the invention aims to solve]

[0006] When performing photon counting using a CMOS image sensor, random readout noise is generated within the amplifier when the amplified voltage is read out. If the readout noise is large, the probability distribution of observed photoelectrons becomes broad. Therefore, it is desirable for the readout noise of each pixel to be small. However, when CMOS image sensors are manufactured, the readout noise of pixels may vary within a certain range. In this case, pixels with high readout noise may experience a decrease in photon counting accuracy.

[0007] One aspect of this disclosure aims to provide a photon counting device that can suppress the decrease in photon counting accuracy. [Means for solving the problem]

[0008] One example of a photon counting device includes a plurality of pixels, each containing a photoelectric conversion element that converts input light into electric charge and an amplifier that amplifies the charge converted by the photoelectric conversion element and converts it into a voltage; an A / D converter that converts the voltages output from the amplifiers of the plurality of pixels into digital values; a first derivation unit that derives a provisional value for the number of photons of each pixel in the plurality of pixels based on the digital values; and a second derivation unit that derives a definitive value for the number of photons of a target pixel, which is one of the plurality of pixels, based on a first probability and a second probability, wherein the first probability is the observed probability for each number of photons in the target pixel based on the probability distribution of the number of photons associated with the photon number distribution of light, and the second probability is the observed probability for each number of photons in the provisional value of the target pixel based on the probability distribution of the number of photons associated with the readout noise of the target pixel.

[0009] In the above photon counting device, based on the magnitude of the digital value corresponding to the amount of charge generated in each pixel, the first derivation unit derives a provisional value of the number of photons in each pixel. For example, in pixels with large readout noise, the error included in the derived provisional value may become large. The second derivation unit derives a determined value of the number of photons when the target pixel indicates the provisional value, based on the probability distribution of the number of photoelectrons associated with the photon number distribution of light and the probability distribution of the number of photoelectrons associated with the readout noise. Thus, the determined value of the number of photons is derived considering the magnitude of the readout noise in the target pixel. Therefore, the influence of the readout noise on the derivation of the determined value can be reduced, and the accuracy of photon counting can be improved.

[0010] In one example, the second derivation unit may calculate the probability for each number of photoelectrons when the target pixel indicates the provisional value by the product of the first probability and the second probability, and determine the determined value based on the calculated probability. In this configuration, by setting the number of photoelectrons showing the maximum value among the probabilities for each number of photoelectrons when the target pixel indicates the provisional value as the determined value, the most probable number of photons can be obtained.

[0011] The probability distribution of the number of photoelectrons associated with the photon number distribution of light in one example may be any one of a Poisson distribution, a super-Poisson distribution, a sub-Poisson distribution, the photon number distribution shown by a photon number squeezed state, the photon number distribution shown by an entangled photon state, the photon number distribution of a multimode squeezed state, a Bose-Einstein distribution, a log-normal distribution, a uniform distribution, or a mixed distribution. In this configuration, the probability distribution of the number of photoelectrons associated with the photon number distribution of light can be appropriately described.

[0012] The probability distribution of the number of photoelectrons associated with the readout noise of the target pixel in one example may be a normal distribution. In this configuration, the probability distribution of the number of photoelectrons associated with the readout noise can be appropriately described.

[0013] In one example, the second derivation unit may calculate the average value of the provisional values of the peripheral pixels by using, as the peripheral pixels, two or more pixels included in a partial area around the target pixel among the plurality of pixels, and calculate the first probability in consideration of the average value. In this configuration, since the average value of the number of photoelectrons of the peripheral pixels is considered, the reliability of the first probability is enhanced.

[0014] The average value in one example may be a weighted average including the read noise of the peripheral pixels as weights. In this configuration, it is possible to obtain an average value with enhanced reliability of the number of photoelectrons in the peripheral pixels with low read noise.

[0015] The average value in one example may be a weighted average including the distance between each of the target pixel and the peripheral pixels as weights. In this configuration, it is possible to obtain an average value with enhanced reliability of the number of photoelectrons in the peripheral pixels closer to the target pixel.

[0016] The average value in one example may be a weighted average including weights such that the error from the average value of the number of photons of the peripheral pixels is reduced. By using such a weighted average, an improvement in the calculation accuracy of the average value can be expected.

[0017] In one example, the second derivation unit may calculate the average value of the provisional values based on the data of the provisional values in a plurality of frames. By using the provisional values in a plurality of frames in this way, an improvement in the calculation accuracy of the average value can be expected.

[0018] In one example, the second derivation unit may create photon counting data for a plurality of pixels by using the determined value derived from pixels having a read noise greater than or equal to a predetermined value among the plurality of pixels as the target pixels and the provisional values of pixels having a read noise less than the predetermined value among the plurality of pixels. In this configuration, for pixels having a read noise less than the predetermined value, the operation of deriving the observation probability becomes unnecessary.

[0019] In one example, the second derivation unit may create photon counting data for multiple pixels using the definitive values ​​derived from target pixels that have provisional values ​​less than a predetermined value, and the provisional values ​​of pixels that have provisional values ​​greater than or equal to the predetermined value. In this configuration, it is unnecessary to perform calculations to derive the observation probability for pixels that have provisional values ​​greater than or equal to the predetermined value.

[0020] One example of a second derivation unit may have a noise map showing the readout noise of each of the multiple pixels. That is, the second derivation unit may derive a second probability by referring to data including the noise map.

[0021] One example of a photon counting method comprises: deriving a provisional value for the number of photons in a plurality of pixels based on digital values ​​corresponding to a plurality of pixels output from a two-dimensional image sensor having a plurality of pixels; and deriving a definitive value for the number of photons in a target pixel, which is one of the plurality of pixels, based on a first probability and a second probability. The definitive value is derived by: firstly, determining the observation probability for each number of photoelectrons in the target pixel based on the probability distribution of the number of photoelectrons associated with the photon number distribution of light; and secondly, determining the observation probability for each number of photoelectrons in the provisional value of the target pixel based on the probability distribution of the number of photoelectrons associated with the readout noise of the target pixel.

[0022] In the photon counting method described above, a provisional value for the number of photons at each pixel is derived based on the magnitude of the digital value corresponding to the amount of charge generated at each pixel. For example, in pixels with large readout noise, the error included in the derived provisional value may be large. Furthermore, the definitive value for the number of photons when the target pixel shows a provisional value is derived based on the probability distribution of the number of photoelectrons associated with the photon number distribution of light and the probability distribution of the number of photoelectrons associated with the readout noise. In this way, the definitive value for the number of photons is derived taking into account the magnitude of the readout noise at the target pixel. Therefore, the influence of readout noise on the definitive value can be reduced, and the accuracy of photon counting can be improved.

[0023] One way to derive a definitive value is to calculate the probability for each photoelectron count when the target pixel shows a provisional value by multiplying the first probability and the second probability, and then determine the definitive value based on the calculated probabilities. In this configuration, the most likely number of photons can be obtained by setting the number of photoelectrons that shows the maximum value among the probabilities for each photoelectron count when the target pixel shows a provisional value as the definitive value.

[0024] To derive a definitive value as an example, one of the following distributions may be used as the probability distribution of the number of photoelectrons associated with the photon number distribution of light: Poisson distribution, hyper-Poisson distribution, sub-Poisson distribution, photon number distribution shown in photon-squeezed states, photon number distribution shown in quantum-entangled photon states, photon number distribution in multimode-squeezed states, Bose-Einstein distribution, log-normal distribution, uniform distribution, or mixture distribution. This configuration allows for an appropriate description of the probability distribution of the number of photoelectrons associated with the photon number distribution of light.

[0025] To derive a definitive value as an example, a normal distribution may be used as the probability distribution of the number of photoelectrons associated with the readout noise of the target pixel. In this configuration, the probability distribution of the number of photoelectrons associated with the readout noise can be appropriately described.

[0026] To derive a definitive value for one example, two or more pixels included in a portion of the area surrounding the target pixel are designated as peripheral pixels, the average value of the provisional values ​​in the peripheral pixels is calculated, and the first probability is calculated considering the average value. In this configuration, the reliability of the first probability is increased because the average number of photoelectrons in the peripheral pixels is taken into consideration.

[0027] To derive a definitive value as an example, a weighted average that includes the readout noise of peripheral pixels as a weighting may be used as the average value. In this configuration, an average value with increased reliability of the photoelectron count in peripheral pixels with low readout noise can be obtained.

[0028] To derive a definitive value for an example, a weighted average may be used, which includes the distance between the target pixel and surrounding pixels as a weighting factor. In this configuration, an average value can be obtained with increased reliability of the photoelectron count of surrounding pixels closer to the target pixel.

[0029] To derive a definitive value for a single example, a weighted average may be used, which includes weights that minimize the error with the average number of photons of surrounding pixels. By using such a weighted average, an improvement in the accuracy of calculating the average value can be expected.

[0030] To derive a definitive value for a single example, one can calculate the average of provisional values ​​based on provisional data from multiple frames. By using provisional values ​​from multiple frames in this way, an improvement in the accuracy of the average calculation can be expected.

[0031] One example method may further include creating photon counting data for multiple pixels using a determined value derived from a target pixel having readout noise above a predetermined value, and a provisional value from a pixel having readout noise below a predetermined value. In this configuration, it is unnecessary to perform calculations to derive observation probabilities for pixels with readout noise below a predetermined value.

[0032] One example method may further include creating photon counting data for multiple pixels using the definitive values ​​derived from target pixels that have provisional values ​​less than a predetermined value, and the provisional values ​​of pixels that have provisional values ​​greater than or equal to the predetermined value. In this configuration, it is unnecessary to perform calculations to derive observation probabilities for pixels that have provisional values ​​greater than or equal to the predetermined value.

[0033] Deriving a definitive value may involve referring to a noise map that shows the readout noise of each of the multiple pixels. For example, the second probability may be derived based on data that includes a noise map.

[0034] One example of a photon counting processing program is a program that causes a computer to perform photon counting processing based on digital values ​​corresponding to multiple pixels output from a two-dimensional image sensor having multiple pixels. The program causes the computer to perform a first derivation process that derives a provisional value for the number of photons of each pixel in the multiple pixels based on the digital values, and a second derivation process that derives a definitive value for the number of photons of a target pixel, which is one of the multiple pixels, based on a first probability and a second probability. The first probability is the observed probability for each number of photoelectrons in the target pixel, based on the probability distribution of the number of photoelectrons associated with the photon number distribution of light, and the second probability is the observed probability for each number of photoelectrons in the provisional value of the target pixel, based on the probability distribution of the number of photoelectrons associated with the readout noise of the target pixel. [Effects of the Invention]

[0035] According to the photon counting device and photon counting method described above, the decrease in the accuracy of photon counting can be suppressed. [Brief explanation of the drawing]

[0036] [Figure 1] Figure 1 shows the configuration of an example photon counting device. [Figure 2] Figure 2 is a schematic diagram showing a 3x3 pixel arrangement. [Figure 3]Figure 3 shows the probability distribution of the number of photoelectrons. [Figure 4] Figure 4 is a schematic diagram illustrating the derivation of the final value. [Figure 5] Figure 5 shows the probability distribution of the number of photoelectrons. [Figure 6] Figure 6 shows the probability distribution of the number of photoelectrons. [Figure 7] Figure 7 shows the probability distribution of the number of photoelectrons. [Figure 8] Figure 8 shows the probability distribution of the number of photoelectrons. [Figure 9] Figure 9 shows the probability distribution of the number of photoelectrons. [Figure 10] Figure 10 is a comparison of the probabilities shown in Figures 5 through 9. [Figure 11] Figure 11 is a schematic diagram illustrating the derivation of the final value. [Figure 12] Figure 12 shows the probability distribution of the number of photoelectrons. [Figure 13] Figure 13 shows the probability distribution of the number of photoelectrons. [Figure 14] Figure 14 shows the probability distribution of the number of photoelectrons. [Figure 15] Figure 15 shows the probability distribution of the number of photoelectrons. [Figure 16] Figure 16 shows the probability distribution of the number of photoelectrons. [Figure 17] Figure 17 is a schematic diagram illustrating the derivation of the final value. [Figure 18] Figure 18 shows the probability distribution of the number of photoelectrons. [Figure 19] Figure 19 shows the probability distribution of the number of photoelectrons. [Figure 20] Figure 20 shows the probability distribution of the number of photoelectrons. [Figure 21] Figure 21 shows the probability distribution of the number of photoelectrons. [Figure 22] Figure 22 is a flowchart showing the operation of an example photon counting device. [Figure 23]Figure 23 shows the process by which a definitive value is derived from a pixel value. [Figure 24] This is a diagram showing a photon counting processing program. [Figure 25] Figure 25 is a diagram illustrating an example of photon counting results. [Figure 26] Figure 26 is a diagram illustrating an example of photon counting results. [Figure 27] Figure 27 is a schematic diagram showing other forms of peripheral pixels. [Figure 28] Figure 28 is a schematic diagram showing other forms of peripheral pixels. [Figure 29] Figure 29 is a schematic diagram showing other forms of peripheral pixels. [Figure 30] Figure 30 is a schematic diagram showing other forms of peripheral pixels. [Figure 31] Figure 31 is a diagram illustrating another example of a weighted average. [Figure 32] Figure 32 is a diagram illustrating another example of a weighted average. [Figure 33] Figure 33 is a diagram illustrating another example of a weighted average. [Modes for carrying out the invention]

[0037] The embodiments will be described in detail below with reference to the drawings. For convenience, substantially identical elements will be given the same reference numerals, and their descriptions may be omitted. In the following description, photon counting includes both the counting of the number of photoelectrons generated at each pixel of the image sensor, and the counting of the number of photons considering the quantum efficiency (QE) of the image sensor. Such photon counting is also called photon number resolving. In general, photon counting also includes both the detection of photoelectrons generated at each pixel of the image sensor, and the detection of photons incident on each pixel of the image sensor.

[0038] Figure 1 shows the configuration of an example photon counting device. As shown in Figure 1, an example photon counting device comprises a CMOS image sensor 10 as a two-dimensional image sensor and a computer (control device) 20 connected to the CMOS image sensor 10. The CMOS image sensor 10 includes a plurality of pixels 11 and an A / D converter 15. The plurality of pixels 11 are arranged in two dimensions; that is, the plurality of pixels 11 are arranged in the row direction and the column direction. Each pixel 11 has a photodiode (photoelectric conversion element) 12 and an amplifier 13. The photodiode 12 stores photoelectrons generated by the input of photons as electric charge. The amplifier 13 converts the charge stored in the photodiode 12 into a voltage and amplifies the converted voltage. The amplified voltage is transferred to a vertical signal line 16 line by line (row by row) by switching a selection switch 14 of each pixel 11. A CDS (correlated double sampling) circuit 17 is arranged on each vertical signal line 16. The CDS circuit 17 removes noise that varies between pixels and temporarily stores the transferred voltage.

[0039] The A / D converter 15 converts the voltage output from each amplifier 13 in multiple pixels 11 into a digital value. The A / D converter 15 may be provided in each pixel 11. In this embodiment, the A / D converter 15 converts the voltage stored in the CDS circuit 17 into a digital value. The converted digital values ​​are output to the computer 20. For example, the digital values ​​may be sent to a horizontal signal line (not shown) by switching column selection and output to the computer 20. Thus, when a photon is input to each pixel 11 of the CMOS image sensor 10, it outputs a digital value to the computer 20 corresponding to the number of input photons (number of photoelectrons generated). Note that when the voltage amplified by the amplifier 13 is read out, random noise, or readout noise, is generated within the amplifier 13.

[0040] Physically, the computer 20 is composed of storage devices such as RAM and ROM, processors (arithmetic circuits) such as CPUs and GPUs, and communication interfaces. Examples of the computer 20 include personal computers, cloud servers, smart devices (smartphones, tablet terminals, etc.), microcomputers, and FPGAs (field-programmable gate arrays). The computer 20 functions as a storage unit 21, a conversion unit 22, a data processing unit 23, and a control unit 24 by executing programs stored in the storage device using the computer system's processor. The computer 20 may be located inside or outside the camera device, including the CMOS image sensor 10. A display device 25 and an input device 26 may be connected to the computer 20. The display device 25 is, for example, a display capable of displaying photon counting results obtained by the computer 20. The input device 26 may be a keyboard, mouse, etc., for the user to input measurement conditions. The display device 25 and the input device 26 may be touchscreens. The display device 25 and the input device 26 may be included in the computer 20. Furthermore, the display device 25 and the input device 26 may be provided in a camera device including a CMOS image sensor 10.

[0041] The memory unit 21 stores data for converting digital values ​​output from the CMOS image sensor 10 into photon counts. For example, the memory unit 21 stores the gain and offset values ​​of multiple pixels 11 as a lookup table. The memory unit 21 also stores the readout noise of multiple pixels 11 as a lookup table (noise map).

[0042] The digital value [DN] output from the A / D converter 15 described above is given by the following equation (1). Therefore, the offset value [DN] is given as the digital value output when no light is input. In one example, multiple digital values ​​are obtained from multiple dark images acquired by the CMOS image sensor 10 when no light is input, and the offset value is obtained by averaging the acquired digital values ​​for each pixel 11. Also, when obtaining the gain [DN / e] of each pixel 11, multiple frame images are acquired by the CMOS image sensor 10 with sufficient light. Then, the average optical signal value S [DN] and the standard deviation N [DN] of the digital values ​​at each pixel 11 are obtained. The gain is N 2 Since it is expressed as / S, the gain can be derived from the average optical signal value S and the standard deviation N.

number

[0043] Furthermore, readout noise can be defined, for example, as fluctuations in digital values ​​and expressed as a value converted to electron units. Therefore, the readout noise for each pixel 11 may be obtained by acquiring the standard deviation of the digital value for each pixel 11 in multiple (e.g., 100 frames or more) dark images and dividing the acquired standard deviation by the gain of the pixel 11. The offset value, gain, and readout noise for each pixel may be acquired during the manufacturing process of the photon counting device.

[0044] The conversion unit 22 refers to a table stored in the storage unit 21 and converts the digital values ​​for each of the multiple pixels 11 output from the A / D converter 15 into photon numbers (photoelectron numbers). In one example, the photon number can be obtained by dividing the photoelectron number for each pixel 11 by the quantum efficiency. When the quantum efficiency is 100%, the photoelectron number and the photon number are the same.

[0045] The data processing unit 23 creates a two-dimensional image showing the number of photons in each pixel 11 based on the number of photons output from the conversion unit 22. For example, the two-dimensional image may be an image in which each pixel is drawn with brightness corresponding to the number of photons. The created two-dimensional image can be output to the display device 25. The data processing unit 23 may also create a histogram, which is a plot of the number of pixels against the number of photons. The control unit 24 can comprehensively control each function unit of the computer 20 and the CMOS image sensor 10.

[0046] The conversion unit 22 will be described in detail below. In the description of the conversion unit 22, a group of pixels arranged in a 3x3 grid may be referred to as a part of the image sensor composed of multiple pixels. Figure 2 is a schematic diagram showing a group of pixels arranged in a 3x3 grid. In Figure 2, the readout noise corresponding to each pixel 11 constituting the pixel group is "R i This is indicated by the symbol (i indicates the pixel position). The conversion unit 22 can appropriately refer to the gain, offset value, and readout noise of each pixel 11 by referring to the lookup table held by the storage unit 21.

[0047] One example of a conversion unit 22 includes a provisional value derivation unit 22a (first derivation unit) and a final value derivation unit 22b (second derivation unit). The provisional value derivation unit 22a derives a provisional value for the number of photons of each pixel 11 in a plurality of pixels 11 based on the digital value. In the provisional value derivation unit 22a, the number of photoelectrons obtained by subtracting the offset value from the measured digital value and dividing the result by the gain may be derived as a provisional value for the number of photons (first provisional value) for each pixel 11, as shown in equation (2) below. Hereinafter, the first provisional value may be referred to as the pixel value.

number

[0048] Furthermore, the provisional value derivation unit 22a may derive an integer value of the number of photons estimated from the pixel value as a provisional value (second provisional value). Hereinafter, the second provisional value may be referred to as the provisional number of photons. In one example, the provisional number of photons may be obtained by rounding the decimal part of the pixel value. In this case, the pixel value may be converted to the provisional number of photons by setting a predetermined threshold range for the pixel value. For example, the threshold range corresponding to 5 photoelectrons is 4.5e or more and less than 5.5e. Note that in Figure 2, the provisional value (for example, the provisional number of photons) in each pixel 11 constituting the pixel group is "k i This is indicated by the sign (where i indicates the pixel position).

[0049] The fixed value derivation unit 22b derives (determines) a fixed value for the number of photons of each of the multiple pixels 11. For example, the fixed value derivation unit 22b selects one of the multiple pixels 11 as a target pixel and derives a fixed value for the number of photons of that target pixel. By selecting each of the multiple pixels constituting the 2D image sensor as a target pixel, a fixed value for the number of photons of all pixels is derived.

[0050] In this embodiment, the definitive value derivation unit 22b derives a first probability and a second probability, and derives a definitive value for the number of photons in the target pixel based on the derived first and second probabilities. The first probability is the observed probability for each number of photoelectrons in the target pixel, based on the probability distribution of the number of photoelectrons associated with the photon number distribution of light, and is shown by the following equation (3). As shown in equation (3), one example of the first probability is based on the probability distribution of the number of photoelectrons associated with optical shot noise and follows a Poisson distribution.

number

[0051] In equation (3) above, k represents the number of photons and λ represents the average number of photons. That is, the first probability is the probability (observation probability) that the number of photons in the target pixel is observed to be k when the average number of photons in the target pixel is λ, and this is determined for each number of photoelectrons. Note that the number of photons k is a provisional number of photons assumed by the definitive value derivation unit 22b. That is, the number of photons k can be said to be a provisional value (third provisional value) of the number of photons in the target pixel. Hereafter, the third provisional value may be referred to as the assumed number of photons.

[0052] The average number of photons (average value) may be the average of the provisional values ​​of the surrounding pixels. Surrounding pixels can be defined as two or more pixels that are included in a portion of the area surrounding the target pixel, among a group of pixels. In the example of the 3x3 pixel group shown in Figure 2, the central pixel 11c may be defined as the target pixel, and the 3x3 pixel group may be defined as the surrounding pixels. In this case, the average number of photons in the target pixel will be the average of the provisional values ​​of the pixels 11 that make up the surrounding pixels. The provisional value of the target pixel among the surrounding pixels may be an assumed number of photons. That is, in Figure 2, when deriving the average number of photons of pixel 11c, k0 may be an assumed number of photons. The provisional value of the surrounding pixels excluding the target pixel may be either the pixel value or the provisional number of photons. Note that for the target pixel, either the pixel value or the provisional number of photons may be used instead of the assumed number of photons as the provisional value.

[0053] In one example, the definitive value derivation unit 22b may refer to a noise map showing the readout noise of each of the multiple pixels 11 and calculate a weighted average as the average number of photons, which includes the readout noise of the surrounding pixels as a weight. The weight W is based on the readout noise. i (where i indicates the pixel position) is shown, for example, by the following equation (4). That is, one example of weight W i This is the readout noise R i It may be a power of the reciprocal of . In this case, pixels with small readout noise are more likely to have the provisional value reflected in the average number of photons, and pixels with large readout noise are less likely to have the provisional value reflected in the average number of photons. In equation (4), the confidence α is the result of the readout noise being weighted W iThe influence on weight W can be increased or decreased. In other words, the greater the confidence α, the greater the influence of the readout noise. i The impact on this will be significant. In one example, α ≥ 0. However, if the confidence level α becomes too large, it is possible that the correct final value will not be derived. Therefore, in one example, the confidence level α may be less than 20. The confidence level α may be a value that is pre-set in the final value derivation unit 22b, or it may be a value that can be set by the user of the photon counting device 1.

number

[0054] The average photon number λ based on the weighted average is given by the following equation (5).

number

[0055] The second probability is the observation probability for each photoelectron count in the provisional value of the target pixel, based on the probability distribution of photoelectron counts associated with the readout noise of the target pixel, and is shown by equation (6) below. The provisional value of the target pixel may be the pixel value. As shown in equation (6), the second probability follows a normal distribution (Gaussian distribution). In equation (6), x is the pixel value of the target pixel [e], and R is the readout noise of the target pixel [e-rms]. That is, the second probability is the probability (observation probability) that the number of photons of the target pixel is observed to be k in the provisional value of the target pixel (e.g., the pixel value), and is calculated for each photoelectron count.

number

[0056] The definitive value derivation unit 22b calculates the probability for each photoelectron count when the target pixel shows a provisional value based on the product of the first probability and the second probability, and determines the definitive value of the photon count based on the calculated probabilities. That is, in one example, the definitive value derivation unit 22b calculates the probability for each assumed photon count when the target pixel shows a provisional value based on the following equation (7) while changing the assumed photon count of the target pixel, and outputs the value of the assumed photon count with the highest probability as the definitive value of the photon count. The range of assumed photon counts calculated by the definitive value derivation unit 22b may be determined based on the provisional value of the target pixel and the average photon count. For example, the range of assumed photon counts may be the smallest range that includes the provisional value of the target pixel and the average photon count. In this case, the average photon count may be calculated without including the provisional value of the target pixel. Alternatively, for example, the range of assumed photon counts may be from 0 to the maximum value of the provisional value in the surrounding pixels.

number

[0057] In one example, equation (7) above may be transformed as follows for ease of calculation. That is, by taking the logarithm of both sides of equation (7), the following equation (8) is derived.

number

[0058] Since all terms in equation (8) except for the term relating to the number of photons are unnecessary, equation (8) can be approximated as shown in equation (9) below. One example of a definitive value derivation unit 22b can derive a definitive value for the number of photons based on equation (9) below.

number

[0059] Figure 3 shows the probability distribution of the number of photoelectrons. In Figure 3, the probability distribution of the number of photoelectrons is shown when the pixel value of the target pixel is 4.2[e] and the average number of photons of the target pixel is 2.5 photons. In Figure 3, the first probability when the assumed number of photons is 3 is shown by the thick line L1, and the second probability distribution when the assumed number of photons is 3 is shown by the dashed line L2. The probability distribution of the product of the first and second probabilities when the assumed number of photons is 3 is shown by the solid line L3. The second probability when the assumed number of photons is 3 and the pixel value is 4.2[e] is shown by the dashed line L4, so the product of the first and second probabilities when the assumed number of photons is 3 and the pixel value is 4.2[e], i.e., P(3|4.2), is shown by the dashed line L5.

[0060] As described above, the confirmed value derivation unit 22b uses the provisional values ​​of the surrounding pixels as a clue to derive the most probabilistically possible number of photons for the target pixel as the confirmed value of the target pixel. The confirmed value derivation unit 22b will be further explained below using specific numerical values. Three examples will be explained here: an example where the readout noise of the target pixel is large, an example where the readout noise of the target pixel is small, and an example where the amount of light to the 2D image sensor is large. In one example, a group of pixels arranged in 3 rows x 3 columns is described as the surrounding pixels, but below, for the sake of simplicity, the surrounding pixels will be described as being arranged in 1 row x 3 columns. In this case, the central pixel is the target pixel.

[0061] Figure 4 shows an example where the readout noise of the target pixels is large. Figure 4(a) shows the pixel value [e] of each pixel 11. Figure 4(b) shows the provisional photon count of each pixel 11. Figure 4(c) shows the readout noise of each pixel 11. Figure 4(d) shows the weight of each pixel 11 in the weighted average. In the example in Figure 4, the pixel values ​​[e] of the three pixels 11 are "1.1", "4.2", and "0.3", respectively, and the provisional photon counts of the three pixels are derived as "1", "4", and "0", respectively. The readout noise [e-rms] of the three pixels are "0.2", "2.0", and "0.4", respectively. In the illustrated example, the confidence level α is "2", and the weights of the three pixels are "25", "0.25", and "6.25".

[0062] The definitive value derivation unit 22b derives the probability of the assumed number of photons being 4.2[e] based on equation (9) above, while changing the assumed number of photons. Figures 5 to 9 show the probability of the number of photoelectrons in the example of Figure 4. Figures 5 to 9 show the Poisson distribution corresponding to the average number of photons, as well as P(k|x) corresponding to the assumed number of photons k. Figure 5 shows the probability of the number of photoelectrons when the assumed number of photons is "0". In the example of Figure 5, since the assumed number of photons is "0", the average number of photons of the target pixel is approximately 0.79[e]. Also in Figure 5, the probability of the assumed number of photons being "0" when the pixel value is 4.2 is shown at the position of the number of photoelectrons 4.2[e].

[0063] Similarly, Figure 6 shows the probability of the number of photoelectrons when the assumed number of photons is "1". In the example in Figure 6, since the assumed number of photons is "1", the average number of photons for the target pixel is approximately 0.80. Also in Figure 6, the probability that the assumed number of photons is "1" when the pixel value is 4.2 is shown at the position of photoelectron number 4.2[e]. Figure 7 shows the probability of the number of photoelectrons when the assumed number of photons is "2". In the example in Figure 7, since the assumed number of photons is "2", the average number of photons for the target pixel is approximately 0.81. Also in Figure 7, the probability that the assumed number of photons is "2" when the pixel value is 4.2 is shown at the position of 4.2[e]. Figure 8 shows the probability of the number of photoelectrons when the assumed number of photons is "3". In the example in Figure 8, since the assumed number of photons is "3", the average number of photons for the target pixel is approximately 0.82. Also in Figure 8, the probability that the assumed number of photons is "3" when the pixel value is 4.2 is shown at the position of 4.2[e]. Figure 9 shows the probability of the number of photoelectrons when the assumed number of photons is "4". In the example in Figure 9, since the assumed number of photons is "4", the average number of photons for the target pixel is approximately 0.83. Also in Figure 9, the probability that the assumed number of photons is "4" when the pixel value is 4.2 is shown at the position 4.2[e].

[0064] Figure 10 is a magnified comparison of the positions where the number of photoelectrons is 4.2[e] in Figures 5 to 9. As shown in Figure 10, in the example in Figure 4, the probability is highest when the assumed number of photons is "1". The definitive value derivation unit 22b in one example compares the probabilities based on the calculation results of equation (9) and derives 1[e] as the definitive value for the number of photons of the target pixel.

[0065] Figure 11 shows an example where the readout noise of the target pixels is small. Figure 11(a) shows the pixel value [e] of each pixel. Figure 11(b) shows the provisional photon count of each pixel. Figure 11(c) shows the readout noise [e-rms] of each pixel. Figure 11(d) shows the weight of each pixel in the weighted average. In the example in Figure 11, the pixel values ​​[e] of the three pixels are "1.1", "4.2", and "0.3", respectively, and the provisional photon counts of the three pixels are derived as "1", "4", and "0", respectively. The readout noise [e-rms] of the three pixels are "0.2", "0.3", and "0.4", respectively. In the illustrated example, the confidence level α is "2", and the weights of the three pixels are "25", "11.1", and "6.25".

[0066] The definitive value derivation unit 22b derives the probability that the pixel value is 4.2 when the assumed number of photons is obtained, based on equation (9) above, while changing the assumed number of photons. Figures 12 to 16 show the probability of the number of photoelectrons in the example of Figure 11. Figures 12 to 16 show the Poisson distribution corresponding to the average number of photons, as well as P(K0|x) corresponding to the assumed number of photons. In the example of Figure 12, since the assumed number of photons k is "0", the average number of photons of the target pixel is approximately 0.59. Also in Figure 12, the probability that the assumed number of photons is "0" when the pixel value is 4.2 is shown at the position of the photoelectron number 4.2[e].

[0067] Similarly, Figure 13 shows the probability of the number of photoelectrons when the assumed number of photons is "1". In the example in Figure 13, since the assumed number of photons is "1", the average number of photons for the target pixel is approximately 0.85. Also in Figure 13, the probability that the assumed number of photons is "1" when the pixel value is 4.2 is shown at the position of the number of photoelectrons 4.2[e]. Figure 14 shows the probability of the number of photoelectrons when the assumed number of photons is "2". In the example in Figure 14, since the assumed number of photons is "2", the average number of photons for the target pixel is approximately 1.11. Also in Figure 14, the probability that the assumed number of photons is "2" when the pixel value is 4.2 is shown at the position of the number of photoelectrons 4.2[e]. Figure 15 shows the probability of the number of photoelectrons when the assumed number of photons is "3". In the example in Figure 15, since the assumed number of photons is "3", the average number of photons for the target pixel is approximately 1.38. Furthermore, Figure 15 shows the probability that the assumed number of photons is "3" when the pixel value is 4.2, at the position of photoelectron number 4.2[e]. Figure 16 is a diagram showing the probability of the number of photoelectrons when the assumed number of photons is "4". In the example in Figure 16, since the assumed number of photons is "4", the average number of photons for the target pixel is approximately 1.64. Also, in Figure 16, the probability that the assumed number of photons is "4" when the pixel value is 4.2 is shown at the position of photoelectron number 4.2[e].

[0068] As shown in Figures 12 to 16, the probability of the assumed number of photons being the same as the provisional number of photons, "4," is higher than the probability of other assumed number of photons. In the illustrated example, the probability of the assumed number of photons being anything other than "4" is almost zero. Therefore, the definitive value derivation unit 22b derives "4" as the definitive value for the number of photons of the target pixel.

[0069] Figure 17 shows an example of a large amount of light for a 2D image sensor. Figure 17(a) shows the pixel value of each pixel. Figure 17(b) shows the provisional photon count of each pixel. Figure 17(c) shows the readout noise of each pixel. Figure 17(d) shows the weight of each pixel in the weighted average. In the example in Figure 17, the pixel values ​​of the three pixels are "108.4", "92.6", and "95.1", respectively, and the provisional photon counts of the three pixels are derived as "108", "93", and "95", respectively. The readout noise of the three pixels is "0.2", "2.0", and "0.4", respectively. In the illustrated example, the confidence level α is "2", and the weights of the three pixels are "25", "0.25", and "6.25".

[0070] The definitive value derivation unit 22b derives the probability that the pixel value is 4.2 when the assumed number of photons is obtained, based on the above formula, while changing the assumed number of photons. Figures 18 to 21 show the probability of the number of photoelectrons in the example of Figure 17. Figures 18 to 21 show the Poisson distribution corresponding to the average number of photons, as well as P(K0|x) corresponding to the assumed number of photons. In the example of Figure 18, since the assumed number of photons is "92", the average number of photons of the target pixel is approximately 105.29. Also in Figure 18, the probability that the assumed number of photons is "92" when the pixel value is 92.6 is shown at the position of the photoelectron number 92.6[e].

[0071] Similarly, Figure 19 shows the probability of the number of photoelectrons when the assumed number of photons is "93". In the example in Figure 19, since the assumed number of photons is "93", the average number of photons for the target pixel is approximately 105.30. In Figure 19, the probability that the assumed number of photons is "93" when the pixel value is 92.6 is shown at the position of the number of photoelectrons 92.6[e]. Figure 20 shows the probability of the number of photoelectrons when the assumed number of photons is "94". In the example in Figure 20, since the assumed number of photons is "94", the average number of photons for the target pixel is approximately 105.31. In Figure 20, the probability that the assumed number of photons is "94" when the pixel value is 92.6 is shown at the position of the number of photoelectrons 92.6[e]. Figure 21 shows the probability of the number of photoelectrons when the assumed number of photons is "95". In the example in Figure 21, since the assumed number of photons is "95", the average number of photons for the target pixel is approximately 105.32. Figure 21 shows the probability that the assumed photon number is "95" when the pixel value is 92.6, at the position of photoelectron number 92.6[e].

[0072] As shown in Figures 18 to 21, the probability of the assumed number of photons being the same as the provisional number of photons, "93," is higher than the probabilities of other assumed number of photons. Therefore, the definitive value derivation unit 22b derives "93" as the definitive value for the number of photons of the target pixel.

[0073] Figure 22 is a flowchart showing the operation of the photon counting device (photon counting method). In this embodiment, when measurement is started with the photon counting device 1 in operation, first, photons incident on the pixels 11 of the CMOS image sensor 10 are converted into electric charge by the photodiode 12 (step S11). The converted electric charge is then converted into a voltage by the amplifier 13 (step S12). This voltage is converted into a digital value by the A / D converter 15 and output to the computer 20 (step S13). The provisional value derivation unit 22a of the conversion unit 22 derives a provisional value from the digital value based on the gain and offset values ​​of each pixel obtained by referring to the table in the storage unit 21 (step S14). The final value derivation unit 22b derives a final value of the number of photons from, for example, equation (9) based on the provisional value of each pixel and the readout noise (step S15).

[0074] Figure 23(a) shows the readout noise of each pixel 11 in a two-dimensional image sensor having 4 rows x 4 columns of pixels. Figure 23(b) shows the process by which a final value is derived from the pixel value of each pixel. The final value derivation unit 22b determines the final value of the number of photons for each of the multiple pixels 11 using the method described above. That is, the final value derivation unit 22b creates photon counting data composed of the final value of the number of photons in each pixel 11 based on the final value derived for each of the multiple pixels 11 as the target pixel. Note that in Figure 23(b), the pixel value for each pixel 11 is converted to a provisional value and then to a final value. For example, in the case of pixel 11a with a pixel value of 1.2[e], 1[e] is derived as a provisional value and 1[e] is also derived as the final value. Also, in the case of pixels with large readout noise of 1.5[e-rms] and 2.0[e-rms], the provisional value and the final value are different values.

[0075] In the illustrated example, the peripheral pixels are a 3x3 pixel group centered on the target pixel. However, if the target pixel is on the periphery of the 2D image sensor, the pixels included in the 3x3 pixel area centered on the target pixel become the peripheral pixels. If pixel 11a with a pixel value of 1.2[e] is the target pixel, then the four pixels within area R become the peripheral pixels. In this way, the number of photons is measured for each of the multiple pixels. The measurement results (photon counting data) are output to the display device 25 as, for example, image data (step S16).

[0076] Figure 24 shows a recording medium 100 containing a program P1 that causes a computer to perform photon counting. The photon counting processing program P1 stored in the recording medium 100 includes a provisional value derivation module P22a, a final value derivation module P22b, a data processing module P23, and a control module P24. The functions (processes) realized by executing the provisional value derivation module P22a, the final value derivation module P22b, the data processing module P23, and the control module P24 are the same as the functions (processes) of the provisional value derivation unit 22a (first derivation process), the final value derivation unit 22b (second derivation process), the data processing unit 23, and the control unit 24 described above.

[0077] The photon counting processing program P1 is recorded in the program recording area of ​​the recording medium 100. The recording medium 100 is composed of, for example, a CD-ROM, DVD, ROM, or semiconductor memory. The photon counting processing program P1 may also be provided via a communication network as a computer data signal superimposed on a carrier wave.

[0078] As described above, one example of a photon counting device 1 includes a plurality of pixels 11, each containing a photodiode 12 that converts input light into electric charge and an amplifier 13 that amplifies the charge converted by the photodiode 12 and converts it into a voltage; an A / D converter 15 that converts the voltages output from the amplifiers 13 of the plurality of pixels 11 into digital values; a provisional value derivation unit 22a that derives a provisional value for the number of photons in each of the plurality of pixels 11 based on the digital values; and a final value derivation unit 22b that derives a final value for the number of photons in a target pixel, which is one of the plurality of pixels 11, based on a first probability and a second probability. The first probability is the observed probability for each number of photoelectrons in the target pixel, based on the probability distribution of the number of photoelectrons associated with the photon number distribution of light, and the second probability is the observed probability for each number of photoelectrons in the provisional value of the target pixel, based on the probability distribution of the number of photoelectrons associated with the readout noise of the target pixel.

[0079] In the photon counting device 1 described above, the provisional value derivation unit 22a derives a provisional value for the number of photons in each pixel 11 based on the magnitude of a digital value corresponding to the amount of charge generated in each pixel 11. For example, in pixels 11 with large readout noise, the error included in the derived provisional value may be large. The final value derivation unit 22b derives a final value for the number of photons when the target pixel shows a provisional value, based on the probability distribution of the number of photoelectrons associated with the photon number distribution of light and the probability distribution of the number of photoelectrons associated with readout noise. In this way, the final value for the number of photons is derived taking into account the magnitude of the readout noise in the target pixel. Therefore, the influence of readout noise on the derivation of the final value can be reduced, and the accuracy of photon counting can be improved.

[0080] One example of a definitive value derivation unit 22b calculates the probability for each assumed number of photons when the target pixel shows a provisional value by multiplying the first probability and the second probability, and determines the definitive value based on the calculated probabilities. In this configuration, the most likely number of photons can be obtained by setting the assumed number of photons that shows the maximum value among the probabilities for each assumed number of photons when the target pixel shows a provisional value as the definitive value.

[0081] The probability distribution of photoelectron numbers associated with the photon number distribution of light, as an example, is based on a Poisson distribution. This configuration can appropriately describe the probability distribution of photoelectron numbers associated with the photon number distribution of light. Furthermore, the probability distribution of photoelectron numbers associated with readout noise in a target pixel, as an example, is based on a normal distribution. This configuration can appropriately describe the probability distribution of photoelectron numbers associated with readout noise.

[0082] One example of a definitive value derivation unit 22b selects two or more pixels 11 from among multiple pixels 11 that are included in a portion of the area surrounding the target pixel as peripheral pixels, calculates the average value of the provisional values ​​in the peripheral pixels, and calculates a first probability taking the average value into consideration. More specifically, the definitive value derivation unit 22b uses the average value of the provisional values ​​of the peripheral pixels as the average number of photons of the target pixel, and derives the observation probability for each number of photoelectrons in the target pixel based on the probability distribution of the number of photoelectrons associated with the photon number distribution of light. In this configuration, the reliability of the first probability is increased by considering the average value of the number of photoelectrons of the peripheral pixels.

[0083] One example of a definitive value derivation unit 22b has a noise map showing the readout noise of each of the multiple pixels 11. That is, the definitive value derivation unit 22b refers to the noise map as needed. The definitive value derivation unit 22b can derive a second probability by referring to the data including the noise map. For example, the definitive value derivation unit 22b can calculate a weighted average by referring to the noise map.

[0084] One example of a definitive value derivation unit 22b calculates a weighted average that includes the readout noise of peripheral pixels as a weight. In this configuration, an average value can be obtained with increased reliability of the photoelectron count in peripheral pixels with low readout noise.

[0085] One example of an average is a weighted average that includes the distance between the target pixel and surrounding pixels as a weighting factor. With this configuration, it is possible to obtain an average value with increased reliability of the photoelectron count of surrounding pixels that are closer to the target pixel.

[0086] Furthermore, as shown in Figures 11 to 16, when the readout noise of the target pixel is small, the reliability of the provisional value at the target pixel is high, and discrepancies between the provisional value and the final value at the target pixel are less likely to occur. For this reason, in one example, the final value derivation unit 22b may derive the same value as the provisional photon count of the target pixel as the final value of the target pixel when the readout noise of the target pixel is small. That is, the final value derivation unit 22b may create photon counting data for multiple pixels based on the final value derived from the target pixel having readout noise of a predetermined value or more, and the provisional value of the pixel having readout noise less than the predetermined value. In this case, when the target pixel has readout noise less than the predetermined value, it is not necessary to perform the calculation to derive the probability P(k│x). In one example, the final value derivation unit 22b may derive the provisional value as the final value for pixels with readout noise of less than 0.4[e-rms]. In other words, the final value derivation unit 22b may perform the above calculation to derive a final value only for pixels where the readout noise is 0.4 [e-rms] or greater. The final value derivation unit 22b can determine which pixels will have their provisional values ​​derived as final values ​​and which pixels will have the above calculation performed to derive the final values ​​performed by referring to the noise map.

[0087] Furthermore, as shown in Figures 17 to 21, when the provisional value at the target pixel is large, the optical shot noise is dominant over the readout noise, so it is highly likely that the provisional value represents the true number of photons. For this reason, in one example, the final value derivation unit 22b may create photon counting data for multiple pixels based on the final value derived from the target pixel having a provisional value less than a predetermined value among multiple pixels, and the provisional value of the pixel having a provisional value greater than or equal to the predetermined value among multiple pixels. In this case, the calculation to derive the probability P(k│x) is not required for pixels having a provisional value greater than or equal to a predetermined value (hereinafter sometimes referred to as the set value) where optical shot noise is dominant. In one example, the final value derivation unit 22b may derive the provisional value as the final value for pixels where the provisional value is greater than or equal to the set value. In other words, the final value derivation unit 22b may perform the above calculation to derive the final value only for pixels where the provisional value is less than the set value. Furthermore, the confirmed value derivation unit 22b may set a range of assumed photon numbers such that, in effect, provisional values ​​are derived as confirmed values ​​for pixels where the provisional value is equal to or greater than the set value. That is, the confirmed value derivation unit 22b may set a range of assumed photon numbers that includes the provisional value of the target pixel in addition to the range from 0 to the set value. For example, if the set value is 10[e], in the example of Figure 17, the range of assumed photon numbers will be "0, 1, 2, ..., 8, 9, 10, 93".

[0088] Figures 25 and 26 show an example of the output results of a photon counting device. Figure 25(a) is an image formed as a result of photon counting by an EMCCD (Electron Multiplying Charge Coupled Device). The image has brightness corresponding to the number of photons measured. Figure 25(b) is an image formed based on digital values ​​from the photon counting device described above. The image has brightness corresponding to the digital values. Figure 25(c) is an image formed based on provisional values ​​from the photon counting device described above. The image has brightness corresponding to the provisional values. Figure 25(d) is an image formed based on final values ​​from the photon counting device described above. The image has brightness corresponding to the final values. In Figure 25, the background light is 0 [photon / pix / frame], the signal is 1 [photon / pix / frame], the exposure time is 200 [ms] (5 [fps]), and the wavelength is 532 [nm]. In Figure 25, numbers, squares, and combinations of three rectangles are drawn using signals. The confidence level α is 10.

[0089] In Figure 25(b), noise is present throughout the image, particularly due to readout noise. The signal portion in Figure 25(c) is more clearly rendered than the EMCCD signal portion shown in Figure 25(a). This is thought to be due to the EMCCD having amplification noise. On the other hand, the background portion in Figure 25(c) contains more noise than the background portion of the EMCCD. This is thought to be due to the inclusion of pixels with high readout noise among the multiple pixels constituting the 2D image sensor. In Figure 25(d), which shows an image reflecting the final values, the clarity of the signal portion is the same as in Figure 25(c). On the other hand, the noise in the background portion of Figure 25(d) is less than the noise in the background portion of Figure 25(c), and the amount of noise is comparable to that of the EMCCD background.

[0090] Figure 26 shows the dark images obtained by the photon counting device. Figure 26(a) is an image formed based on the digital values ​​obtained by the photon counting device described above. Figure 26(b) is an image formed based on the provisional values ​​obtained by the photon counting device described above. Figure 26(c) is an image formed based on the final values ​​obtained by the photon counting device described above.

[0091] In Figure 26(a), noise is present throughout the image, particularly due to readout noise. In Figure 26(b), a significant amount of noise is observed, mainly due to pixels with high readout noise. In Figure 26(c), which shows the image reflecting the final values, the amount of noise is reduced compared to Figure 26(b), which shows the image reflecting the provisional values.

[0092] Although the embodiments have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments.

[0093] For example, we have shown an example where peripheral pixels are formed by a 3x3 pixel group centered around the target pixel, but the configuration of peripheral pixels can be determined arbitrarily. Figures 27, 28, 29, and 30 show other examples of peripheral pixels. In each figure, the target pixel 11c and the other pixels 11 are shown.

[0094] Figure 27 shows an example where the surrounding pixels are arranged in a square shape. The surrounding pixels shown in Figure 27(a) are composed of a 3x5 pixel group. The surrounding pixels shown in Figure 27(b) are composed of a 5x5 pixel group. In the illustrated examples, the central pixel 11c is the target pixel, but any other pixel 11 may be the target pixel. The surrounding pixels may be formed by a group of pixels arranged in an NxM or MxN grid, where N is an integer greater than or equal to 1 and M is an integer greater than or equal to 2.

[0095] Figure 28 shows an example of a group of pixels where the peripheral pixels are arranged symmetrically along both the line and the point. The peripheral pixels shown in Figure 28(a) are composed of five pixels forming a cross shape. The peripheral pixels shown in Figure 28(b) are composed of eleven pixels forming a cross shape. In Figures 28(a) and 28(b), the target pixel 11c is positioned at the intersection of pixels arranged in the row direction (pixel row) and pixels arranged in the column direction (pixel column), but the target pixel may be any other pixel 11. Also, although an example is shown where one pixel row and one pixel column intersect, one or more pixel rows and one or more pixel columns may intersect each other.

[0096] The peripheral pixels shown in Figure 28(c) have a shape in which the pixels at the four corners of a pixel group composed of squares (5 rows x 5 columns) have been removed. The peripheral pixels shown in Figure 28(d) have a shape in which the pixels around the four corners of a pixel group composed of squares (5 rows x 5 columns) have been removed. The peripheral pixels shown in Figure 28(e) have a shape in which the pixels around the four corners of a pixel group composed of squares (7 rows x 7 columns) have been removed. The peripheral pixels shown in Figure 28(f) have a shape in which the pixels around the four corners of a pixel group composed of squares (7 rows x 5 columns) have been removed.

[0097] The peripheral pixels shown in Figure 28(g) are composed of pixels 11 corresponding to the diagonal positions of a group of pixels arranged in a rectangular shape (3 rows x 3 columns). The peripheral pixels shown in Figure 28(h) are composed of pixels 11 corresponding to the diagonal positions of a group of pixels arranged in a rectangular shape (5 rows x 5 columns).

[0098] Figure 29 shows examples of pixel groups in which peripheral pixels are configured symmetrically along both lines and points, including pixels that are spaced apart from each other. In Figure 29(a), the peripheral pixels are composed of nine spaced-apart pixels 11 in a 5x5 pixel group. In Figure 29(b), the peripheral pixels are composed of 16 peripheral pixels 11 and a central pixel 11c in a 5x5 pixel group. In Figure 29(b), the peripheral pixels are composed of pixels at the four corners and a cross-shaped group of pixels including the central pixel 11c in a 5x5 pixel group. In Figure 29(b), the peripheral pixels are composed of the pixel group in the first row, the pixel group in the third row, and the pixel group in the fifth row in a 5x7 pixel group.

[0099] The peripheral pixels shown in Figure 30(a) are composed of pixels 11 arranged to form a swastika shape within a rectangular (5x5) pixel group. The peripheral pixels shown in Figure 30(b) are composed of a triangularly formed pixel group. In this example, one of the vertices of the triangle is the target pixel, but the target pixel may be any other pixel 11. The peripheral pixels shown in Figure 30(c) are composed of pixels 11 that form a spiral shape within a rectangular (5x5) pixel group. The peripheral pixels shown in Figure 30(d) are multiple diagonally consecutive pixel groups within a rectangular (7x7) region, including multiple pixel groups that are spaced apart from each other. In the illustrated example, the pixels 11 at the four corners are included in the peripheral pixels.

[0100] The peripheral pixels shown in Figures 27 to 30 above are merely examples. That is, the configuration of peripheral pixels is not limited to the examples in Figures 27 to 30. For example, each peripheral pixel illustrated in Figures 27 to 30 may be expanded based on regularity. For example, expansion includes increasing the number of consecutive pixels in the row direction, increasing the number of consecutive pixels in the column direction, increasing the number of consecutive pixels in the diagonal direction, etc. Also, each peripheral pixel illustrated in Figures 27 to 30 may be superimposed on each other. When superimposing different peripheral pixels, one or both peripheral pixels may be expanded. Furthermore, peripheral pixels that have a different shape when flipped vertically or horizontally may have the original peripheral pixel and the flipped peripheral pixel superimposed on each other.

[0101] Figure 31 illustrates another example of weighted average. Figure 31 shows a noise map based on the readout noise described in the above embodiment and a map of weights based on the noise map (hereinafter sometimes referred to as noise weights). A modified version of Figure 31 includes distance weights as weights in the weighted average. Distance weights are weights for the weighted average based on the distance between the target pixel and each of the surrounding pixels. In one example, the distance weights are set based on the distance between the center of the target pixel 11c and the centers of each of the surrounding pixels, with the weight increasing as the distance decreases. For example, a mask used as a Gaussian filter may be used as the distance weight. In the illustrated example, the product of the noise weight and the distance weight is used for weighting the weighted average.

[0102] Figure 32 illustrates yet another example of weighted average. Figure 32 shows the noise map due to the readout noise described in the above embodiment and the weight map based on the noise map. Another modification of Figure 32 includes 1 / 0 weights as weights in the weighted average. 1 / 0 weights are weightings where the weight elements are the values ​​of 1 or 0 set for each of the surrounding pixels. In this example, the product of the noise weight and the 1 / 0 weight is used for weighting the weighted average. Therefore, for pixels where the element of the 1 / 0 weight is 1, the noise weight is used directly for weighting the weighted average. On the other hand, for pixels where the element of the 1 / 0 weight is 0, the weighting of the weighted average becomes 0. That is, the provisional values ​​of pixels where the element of the 1 / 0 weight is 0 are not used in the calculation of the average number of photons. For example, a defective pixel contained within multiple pixels constituting a 2D image sensor may be detected, and the 1 / 0 weight for the detected defective pixel may be set to 0. In this case, the output of the defective pixel is not used in the calculation of the average number of photons.

[0103] Furthermore, the 1 / 0 weights can be used to form peripheral pixels of arbitrary shapes, as shown in Figure 33. Figure 33 shows a noise map, a weight map based on the noise map, and the 1 / 0 weights. In the 1 / 0 weights of Figure 33, the weights of the five pixels forming a cross shape are 1, and the weights of the remaining four pixels are zero. In this case, the product of the noise weight and the 1 / 0 weight is used for weighting the weighted average, so that the peripheral pixels essentially take on a shape similar to the peripheral pixels shown in Figure 28(a). In this way, the shape of the peripheral pixels can be arbitrarily set by setting the 1 / 0 weight of any pixel within a rectangular region to 1 and setting the 1 / 0 weight of the remaining pixels to zero.

[0104] Furthermore, while the various embodiments described above show examples in which the first probability is derived based on the probability distribution of photoelectrons associated with optical shot noise, represented by a Poisson distribution, the first probability only needs to be derived based on the probability distribution of photoelectrons associated with the photon number distribution of light.

[0105] For example, if the probability distribution of photoelectrons associated with the photon number distribution of light can be estimated based on the type of light source, the first probability may be derived based on the probability distribution corresponding to the light source. As an example, if the light source is a non-coherent light source such as an LED, or a thermophoton source, the first probability may be derived based on a super-poissonian, which is a photon number distribution where the fluctuation of the photon number is greater than that of the Poisson distribution. Also, if the light source is a quantum light source, the first probability may be derived based on a sub-poissonian, which is a photon number distribution where the fluctuation of the photon number is smaller than that of the Poisson distribution. Furthermore, in this case, the first probability may be derived based on the photon number distribution shown by a photon-squeezed state (e.g., a Fock state) such as a single-photon source, or based on the photon number distribution shown by a quantum-entangled photon state (e.g., a NOON state) generated by a spontaneous parametric down conversion (SPDC). Furthermore, in complex photon states using quantum light sources, the first probability may be derived based on a complex photon number distribution generated by the combination of modes (i.e., the photon number distribution of multi-mode squeezed states). Also, if the light source is a thermal light source or a pseudo-thermal light source, the first probability may be derived based on the Bose-Einstein distribution. In addition, the first probability may be derived based on a log-normal distribution with a tail that extends longer towards larger values, a uniform distribution where the probability is uniform for each photon number, or a mixture distribution which is a combination of multiple photon number distributions.

[0106] Also, the weights used when calculating the average photon number using the weighted average are not limited to the examples of the above embodiments. When using a weight that reduces the error between the average photon number obtained by weighted average and the true average photon number as the weight for calculating the average photon number, the weights obtained as follows may be used. Note that the true average photon number may be the arithmetic mean of the true photon numbers of the surrounding pixels.

[0107] Average photon number λ by weighted average * When obtaining the weight w that reduces the error between and the true average photon number λ, λ * The expected value E[(λ * -λ) 2 , it is only necessary to calculate w that minimizes it. First, find the expected value E[λ * . The pixel value x follows the probability distribution p(x) shown in Equation (10).

Equation

[0108] Calculating the expected value based on this probability distribution gives E[λ * =λ, and the expected value of λ * coincides with λ regardless of the weight w. Subsequently, when finding E[(λ * -λ) 2 , the following Equation (11) is derived.

Equation

[0109] Find the weight W i that minimizes Equation (11). Differentiating with respect to w j and setting it to zero results in Equation (12).

Equation

[0110] Furthermore, writing it out for j = 0 gives Equation (13). ​​

number

[0111] Taking the difference between the two sides for the case where j≠0, we obtain equation (14), and thus derive equation (15).

number

number

[0112] Here, by setting w0 as in equation (16), equation (17) holds for all i. In this case, the weighted average number of photons λ * This is shown by equation (18).

number

number

number

[0113] Note that equation (17) contains the true average number of photons λ, so it cannot be calculated directly using equation (17). Therefore, in one example, we assume that the average number of photons calculated as the unweighted average of surrounding pixels is λ, and we derive w based on equation (17). i This can also be used as a weight.

[0114] Furthermore, w, which was derived based on equation (17), i This can be solved in a self-consistent manner. That is, the derived weights w i By substituting this into equation (18), we obtain the average number of photons (first step), and using this average number of photons, we obtain the weight w from equation (17). i The process of deriving (second step) can be repeated until convergence occurs. Alternatively, the weighted average of the average photon number λ *We can also take the solution to equation (19) as the average photon number, expecting that it approximates the true average photon number λ. This can be solved using the fixed-point theorem, with equation (20) when the function on the right-hand side is a contraction mapping. As described above, by using a weighted average that includes weights in the weighting that reduce the error with the average value of the true photon number of surrounding pixels, we can expect to improve the accuracy of calculating the average photon number.

number

number

[0115] Furthermore, the average number of photons for the target pixel may be derived based on provisional data from multiple frames. That is, the final value derivation unit 22b may acquire provisional data for multiple pixels for multiple frames and derive the average number of photons based on the acquired data. For example, the final value derivation unit 22b may derive the average number of photons for the target pixel for each acquired frame and calculate the first probability using the average value of the derived average number of photons as λ. Alternatively, the final value derivation unit 22b may derive the average number of photons for the target pixel using the provisional data from multiple acquired frames as a single population and calculate the first probability using the derived average number of photons as λ. Alternatively, the final value derivation unit 22b may calculate the average value of the provisional values ​​for each pixel between acquired frames and derive the average number of photons using this average value as the provisional value for each pixel. As described above, by calculating the average number of photons based on provisional data from multiple frames, an improvement in the accuracy of calculating the average number of photons can be expected.

[0116] Furthermore, the definitive value derivation unit 22b may derive the number of photons that is considered to minimize the error with the true number of photons as a definitive value. That is, the definitive value derivation unit 22b may derive the expected value of the number of photons as a definitive value. For example, the definitive value derivation unit 22b may derive the expected value of the number of photons of the target pixel based on the first and second probabilities, using the first probability as the observation probability for each number of photoelectrons in the target pixel based on the probability distribution of the number of photoelectrons associated with the readout noise of the target pixel, and the second probability as the observation probability for each number of photoelectrons in the provisional value of the target pixel. For example, if the probability for each assumed number of photons when the target pixel shows a provisional value is shown by equation (21), the expected value of the number of photons k exp This is shown by equation (22) below. The range of the assumed number of photons k calculated in the definitive value derivation unit 22b may be the data range of the probability distribution of the number of photons.

number

number

[0117] 1...Photon counting device, 11...Pixel, 12...Photodiode (photoelectric conversion element), 13...Amplifier, 15...A / D converter, 21...Storage unit, 22a...Provisional value derivation unit (first derivation unit), 22b...Definitive value derivation unit (second derivation unit).

Claims

1. A plurality of pixels, each including a photoelectric conversion element that converts input light into electric charge and an amplifier that amplifies the charge converted by the photoelectric conversion element and converts it into a voltage, A / D converter that converts the voltage output from the amplifiers of the plurality of pixels into a digital value, A first derivation unit that converts the digital value into a provisional value of the number of photons for each pixel in the plurality of pixels using a predetermined threshold set for the digital value, The system includes a second derivation unit that derives a definitive value for the number of photons in a target pixel, which is one of the plurality of pixels, based on a first probability and a second probability, The first probability is the observed probability for each number of photoelectrons in the target pixel, based on the probability distribution of the number of photoelectrons associated with the photon number distribution of the light. A photon counting device in which the second probability is the observed probability for each number of photoelectrons in the provisional value of the target pixel, based on the probability distribution of the number of photoelectrons associated with the readout noise of the target pixel.

2. The photon counting apparatus according to claim 1, wherein the second derivation unit calculates the probability for each number of photoelectrons when the target pixel shows the provisional value by product of the first probability and the second probability, and determines the final value based on the calculated probability.

3. The photon counting apparatus according to claim 1 or 2, wherein the probability distribution of the number of photoelectrons associated with the photon number distribution of the light is a Poisson distribution, a hyper-Poisson distribution, a sub-Poisson distribution, a photon number distribution shown by a photon number squeezed state, a photon number distribution shown by a quantum entangled photon state, a photon number distribution of a multimode squeezed state, a Bose-Einstein distribution, a log-normal distribution, a uniform distribution, or a mixture distribution.

4. The photon counting apparatus according to any one of claims 1 to 3, wherein the probability distribution of the number of photoelectrons associated with the readout noise of the target pixel is a normal distribution.

5. The photon counting apparatus according to any one of claims 1 to 4, wherein the second derivation unit takes two or more pixels included in a part of the area surrounding the target pixel as peripheral pixels, calculates the average value of the provisional values ​​in the peripheral pixels, and calculates the first probability taking the average value into consideration.

6. The photon counting apparatus according to claim 5, wherein the average value is a weighted average that includes the readout noise of the surrounding pixels as a weight.

7. The photon counting apparatus according to claim 5 or 6, wherein the average value is a weighted average that includes the distance between the target pixel and each of the surrounding pixels as a weight.

8. The photon counting apparatus according to claim 5, wherein the average value is a weighted average that includes weights that reduce the error with the average value of the number of photons of the surrounding pixels.

9. The photon counting apparatus according to any one of claims 5 to 8, wherein the second derivation unit calculates the average value of the provisional values ​​based on the provisional value data in multiple frames.

10. The photon counting apparatus according to any one of claims 1 to 9, wherein the second derivation unit creates photon counting data for the plurality of pixels using the determined value derived from the pixels having the readout noise of a predetermined value or more among the plurality of pixels as the target pixels, and the provisional value of the pixels having the readout noise of a predetermined value or less among the plurality of pixels.

11. The photon counting apparatus according to any one of claims 1 to 10, wherein the second derivation unit creates photon counting data for the plurality of pixels using the determined value derived from the target pixels, which are the pixels among the plurality of pixels that have the provisional value less than a predetermined value, and the provisional value of the pixels among the plurality of pixels that have the provisional value greater than or equal to the predetermined value.

12. The photon counting apparatus according to any one of claims 1 to 11, wherein the second derivation unit has a noise map showing the readout noise of each of the plurality of pixels.

13. Converting a digital value corresponding to a plurality of pixels output from a two-dimensional image sensor having a plurality of pixels into a provisional value of the number of photons for each of the plurality of pixels, using a predetermined threshold value set for the digital value of the plurality of pixels, The method comprises deriving a definite value for the number of photons in a target pixel, which is one of the plurality of pixels, based on a first probability and a second probability, To derive the aforementioned confirmed value, As the first probability, the observation probability for each number of photoelectrons in the target pixel is determined based on the probability distribution of the number of photoelectrons associated with the photon number distribution of light, A photon counting method, comprising determining the observation probability for each number of photoelectrons in the provisional value of the target pixel based on the probability distribution of the number of photoelectrons associated with the readout noise of the target pixel, as the second probability.

14. The photon counting method according to claim 13, wherein the definitive value is derived by calculating the probability for each number of photoelectrons when the target pixel shows the provisional value by the product of the first probability and the second probability, and the definitive value is determined based on the calculated probability.

15. The photon counting method according to claim 13 or 14, wherein the determinant value is derived using a Poisson distribution, a hyper-Poisson distribution, a sub-Poisson distribution, a photon number distribution shown by a photon number squeezed state, a photon number distribution shown by a quantum entangled photon state, a photon number distribution of a multimode squeezed state, a Bose-Einstein distribution, a log-normal distribution, a uniform distribution, or a mixture distribution as the probability distribution of the number of photoelectrons associated with the photon number distribution of the light.

16. The photon counting method according to any one of claims 13 to 15, wherein the derivation of the confirmed value utilizes a normal distribution as the probability distribution of the number of photoelectrons associated with the readout noise of the target pixel.

17. The photon counting method according to any one of claims 13 to 16, wherein the derivation of the confirmed value is performed by taking two or more pixels included in a part of the area surrounding the target pixel as peripheral pixels, calculating the average value of the provisional values ​​in the peripheral pixels, and calculating the first probability taking the average value into consideration.

18. The photon counting method according to claim 17, wherein the derivation of the confirmed value is performed by using a weighted average that includes the readout noise of the surrounding pixels as a weighting for the average value.

19. The photon counting method according to claim 17 or 18, wherein the derivation of the confirmed value is performed using a weighted average that includes the distance between the target pixel and each of the surrounding pixels as weights for the average value.

20. The photon counting method according to claim 17, wherein the derivation of the confirmed value is performed using a weighted average in which the weighting includes weights that reduce the error with the average number of photons of the surrounding pixels.

21. The photon counting method according to any one of claims 17 to 20, wherein the derivation of the confirmed value is calculated based on the provisional value data in multiple frames, and the average value of the provisional values ​​is calculated based on the provisional value data in multiple frames.

22. The photon counting method according to any one of claims 13 to 21, further comprising creating photon counting data for the plurality of pixels using the determined value derived from the pixels having the readout noise of a predetermined value or more among the plurality of pixels as the target pixels, and the provisional value of the pixels having the readout noise of a predetermined value or less among the plurality of pixels.

23. A photon counting method according to any one of claims 13 to 21, further comprising creating photon counting data for the plurality of pixels using the determined value derived from the target pixels of the plurality of pixels having the provisional value less than a predetermined value, and the provisional value of the pixels of the plurality of pixels having the provisional value greater than or equal to the predetermined value.

24. The photon counting method according to any one of claims 13 to 23, wherein the determinant value is derived by referring to a noise map showing the readout noise of each of the plurality of pixels.

25. A program that causes a computer to perform photon counting based on digital values ​​corresponding to multiple pixels output from a two-dimensional image sensor having multiple pixels, A first derivation process that converts the digital value into a provisional value for the number of photons of each pixel in the plurality of pixels using a predetermined threshold set for the digital value, A second derivation process that derives a definitive value for the number of photons in a target pixel, which is one of the plurality of pixels, based on the first and second probabilities, Have the computer run it, The first probability is the observed probability for each number of photoelectrons in the target pixel, based on the probability distribution of the number of photoelectrons associated with the photon number distribution of light. A photon counting processing program in which the second probability is the observed probability for each number of photoelectrons in the provisional value of the target pixel, based on the probability distribution of the number of photoelectrons associated with the readout noise of the target pixel.

Images