Processing device and photoelectric conversion system

By combining a storage unit and a correction unit in the processing device, pixel output values ​​are corrected based on array data, solving the problem of low defect correction accuracy leading to image quality degradation in the prior art, and achieving high-precision defect correction.

CN116614725BActive Publication Date: 2026-11-10CANON KK
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310125610.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2022-11-28
Filing Date
2023-02-15
Publication Date
2026-11-10
Estimated Expiration
2043-02-15

AI Technical Summary

Technical Problem

Existing defect correction methods result in image quality degradation and low defect correction accuracy.

Method used

A processing device including a first storage unit, a second storage unit, and a correction unit is used. The pixel output value is corrected by a calculation unit based on the first array data and the second array data. The correction accuracy is improved by using the correction array data and the calculation unit.

Benefits of technology

It improves the accuracy of defect correction, prevents image quality degradation, and ensures the quality of the corrected image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116614725B_ABST
    Figure CN116614725B_ABST
Patent Text Reader

Abstract

A processing apparatus and a photoelectric conversion system are disclosed. One processing apparatus includes: a first storage unit for storing first array data based on output values ​​of a plurality of pixels arranged in an array; a second storage unit having second array data stored therein, the second array data to be used for correction of output values ​​from the plurality of pixels; and a correction unit including a calculation unit that corrects the output value of at least one of the plurality of pixels based on the first array data and the second array data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to processing devices and photoelectric conversion systems. Background Technology

[0002] As a defect correction unit, a known method is to extract pixels whose output level is higher than that of surrounding pixels as defect pixels and replace the output of the defect pixels with the average value of the output of the surrounding pixels.

[0003] However, the correction method discussed in U.S. Patent Application Publication No. 2007-030365 suffers from a problem of deteriorated accuracy in defect correction, which in turn degrades image quality. The present invention was designed in view of the above-mentioned problems, and its purpose is to prevent image quality degradation by improving the accuracy of defect correction. Summary of the Invention

[0004] According to one aspect of the present invention, a processing apparatus includes: a first storage unit for storing first array data based on output values ​​of a plurality of pixels arranged in an array; a second storage unit having second array data stored therein, the second array data being used for correction of output values ​​from the plurality of pixels; and a correction unit including a calculation unit configured to correct the output value of at least one of the plurality of pixels based on the first array data and the second array data.

[0005] Other features of the invention will become clear from the following description of embodiments with reference to the accompanying drawings. Attached Figure Description

[0006] Figure 1 This is a schematic diagram of a photoelectric conversion device that can be used with the processing apparatus according to the embodiment.

[0007] Figure 2 This is a schematic diagram of a photodiode (PD) substrate of a photoelectric conversion device that can be used with a processing apparatus according to an embodiment.

[0008] Figure 3 This is a schematic diagram of a circuit board for a photoelectric conversion device that can be used with a processing apparatus according to an embodiment.

[0009] Figure 4 This illustration shows an example of the configuration of the pixel circuit of a photoelectric conversion device that can be used with the processing apparatus according to an embodiment.

[0010] Figure 5A , Figure 5B and Figure 5C This is a schematic diagram illustrating the driving of a pixel circuit of a photoelectric conversion device that can be used with a processing apparatus according to an embodiment.

[0011] Figure 6 This is an explanatory diagram of a processing apparatus according to a comparative example of the first embodiment.

[0012] Figure 7 This is an explanatory diagram of the processing apparatus according to the first embodiment.

[0013] Figure 8 This is a block diagram of the processing apparatus according to the first embodiment.

[0014] Figure 9 This is a flowchart illustrating the process to be performed by the processing apparatus according to the first embodiment.

[0015] Figure 10 This is an explanatory diagram of the correction array of the processing apparatus according to the first embodiment.

[0016] Figure 11 This is a block diagram of a processing apparatus according to the second embodiment.

[0017] Figure 12 This is a flowchart illustrating the process to be performed by the processing apparatus according to the second embodiment.

[0018] Figure 13 This is a flowchart illustrating the process to be performed by the processing apparatus according to the third embodiment.

[0019] Figure 14 This is a flowchart illustrating the process to be performed by the processing apparatus according to the fourth embodiment.

[0020] Figure 15 This is an explanatory diagram of the processing apparatus according to the fifth embodiment.

[0021] Figure 16 This is a block diagram of a processing apparatus according to the fifth embodiment.

[0022] Figure 17 This is a flowchart illustrating the process to be performed by the processing apparatus according to the fifth embodiment.

[0023] Figure 18 This is a block diagram of a processing apparatus according to the sixth embodiment.

[0024] Figure 19 This is a flowchart illustrating the process to be performed by the processing apparatus according to the sixth embodiment.

[0025] Figure 20 This is an explanatory diagram of the processing apparatus according to the seventh embodiment.

[0026] Figure 21 This is a block diagram of a processing apparatus according to the seventh embodiment.

[0027] Figure 22This is a flowchart illustrating the process to be performed by the processing apparatus according to the seventh embodiment.

[0028] Figure 23 This is a functional block diagram illustrating a photoelectric conversion system including a processing apparatus according to the eighth embodiment.

[0029] Figure 24A and Figure 24B This is a functional block diagram illustrating a photoelectric conversion system including a processing apparatus according to the ninth embodiment.

[0030] Figure 25 This is a functional block diagram of a photoelectric conversion system including the processing apparatus according to the tenth embodiment.

[0031] Figure 26 It is a functional block diagram of a photoelectric conversion system including the processing apparatus according to the eleventh embodiment.

[0032] Figure 27A and Figure 27B The illustration shows a photoelectric conversion system including a processing apparatus according to the twelfth embodiment. Detailed Implementation

[0033] The following embodiments are provided for implementing the technical concept of the invention and are not intended to limit the invention. For clarity, the size and positional relationships of the components illustrated in the figures are sometimes exaggerated. In the following description, the same components are assigned the same reference numerals, and their descriptions are sometimes omitted.

[0034] Hereinafter, embodiments of the invention will be described in detail with reference to the accompanying drawings. In the following description, terms indicating specific directions and positions (e.g., "up," "down," "right," "left," and other terms including these terms) will be used as needed. The use of these terms will aid in understanding the embodiments described with reference to the accompanying drawings. The scope of the invention is not limited by the meaning of these terms.

[0035] In this specification, a "plan view" refers to a view taken from a direction perpendicular to the light-incident surface of the semiconductor layer. A cross-section refers to a surface in a direction perpendicular to the light-incident surface of the semiconductor layer. When the light-incident surface of the semiconductor layer appears rough under a microscope, the plan view is defined based on the macroscopically observed light-incident surface of the semiconductor layer.

[0036] In the following description, the anode of the avalanche photodiode (APD) is set to a fixed potential, and the signal is extracted from the cathode side. Therefore, the semiconductor region with the same polarity as the signal charge, considered as the majority carrier, is an N-type semiconductor region, and the semiconductor region with the different polarity from the signal charge, considered as the majority carrier, is a P-type semiconductor region.

[0037] The present invention can be implemented even when the cathode of the APD is set to a fixed potential and the signal is taken from the anode side. In this case, the semiconductor region of the first conductivity type, where the polarity of the charge is the same as that of the signal charge, is a P-type semiconductor region, and the semiconductor region of the second conductivity type, where the charge is the majority carrier, is an N-type semiconductor region, where the polarity of the charge is different from that of the signal charge, is also considered. The following description is given with one node of the APD set to a fixed potential, but the potentials of both nodes can be variable.

[0038] In this specification, when the term "impurity concentration" is used, it refers to the net impurity concentration obtained by subtracting the amount compensated by impurities of the opposite conductivity type. In short, "impurity concentration" refers to the net doping concentration. Regions where the impurity concentration of P-type additives is higher than that of N-type additives are P-type semiconductor regions. Conversely, regions where the impurity concentration of N-type additives is higher than that of P-type additives are N-type semiconductor regions.

[0039] Reference Figures 1 to 5A , Figure 5B and Figure 5C The following describes a configuration and driving method of a photoelectric conversion device common to embodiments of the invention, which can be used with a processing device according to an embodiment of the invention. A description of a processing device disposed externally to the photoelectric conversion device will be given below, but the processing device may be included, for example, within the photoelectric conversion device.

[0040] Figure 1 This is a diagram illustrating the configuration of a stacked photoelectric conversion device 100 according to an embodiment of the present invention.

[0041] The photoelectric conversion device 100 includes two stacked substrates electrically connected to a sensor substrate 11 and a circuit substrate 21. The sensor substrate 11 includes a first semiconductor layer containing a photoelectric conversion element 102, which will be described below, and a first wiring structure. The circuit substrate 21 includes a second semiconductor layer containing circuitry (e.g., a signal processing unit 103, which will be described below), and a second wiring structure. The photoelectric conversion device 100 includes a second semiconductor layer, a second wiring structure, a first wiring structure, and a first semiconductor layer stacked in this order. The photoelectric conversion device described in each embodiment is a back-illuminated photoelectric conversion device that receives light entering from a first surface and includes a circuit substrate disposed on a second surface.

[0042] In the following description, sensor substrate 11 and circuit substrate 21 will be described as diced chips, but sensor substrate 11 and circuit substrate 21 are not limited to such chips. For example, each substrate can be a wafer. Alternatively, the substrates can be stacked in wafer state and then diced, or they can be diced into chips and then bonded by stacking the chips.

[0043] A pixel region 12 is arranged on a sensor substrate 11, and a circuit region 22 for processing the signal detected in the pixel region 12 is arranged on a circuit substrate 21.

[0044] Figure 2 This is a diagram illustrating an example arrangement of the sensor substrate 11. Pixels 101, each including a photoelectric conversion element 102 containing an APD, are arranged in a two-dimensional array in the plan view, forming a pixel region 12.

[0045] Pixel 101 is typically the pixel used to form an image. However, when pixel 101 is used as a time-of-flight (TOF) sensor, it is not always necessary to form an image. In other words, pixel 101 can be a pixel used to measure the time of light arrival and also to measure the amount of light.

[0046] Figure 3 This is a configuration diagram of the circuit board 21. The circuit board 21 includes components for processing... Figure 2 The diagram shows a photoelectric conversion element 102, a signal processing unit 103 for the photoelectric converted charge, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit unit 111, a signal line 113, and a vertical scanning circuit unit 110.

[0047] Figure 2 The photoelectric conversion element 102 shown in the figure and Figure 3 The signal processing unit 103 shown in the figure is electrically connected via connection lines set for each pixel.

[0048] Vertical scanning circuit unit 110 receives control pulses supplied from control pulse generation unit 115 and supplies control pulses to each pixel. Logic circuits such as shift registers or address decoders are used as vertical scanning circuit unit 110.

[0049] The signal output from the photoelectric conversion element 102 of the pixel is processed by the signal processing unit 103. A counter and a memory are set in the signal processing unit 103, and the digital value is stored in the memory.

[0050] The horizontal scanning circuit unit 111 inputs control pulses to the signal processing unit 103 to sequentially select each column in order to read signals from the memory storing digital signals of each pixel.

[0051] The signal processing unit 103 outputs a signal from the pixel on the selected column that has been selected by the vertical scanning circuit unit 110 to the signal line 113.

[0052] The signal output to signal line 113 is output to a recording unit or signal processing unit located outside the photoelectric conversion device 100 via output circuit 114.

[0053] exist Figure 2 In this system, photoelectric conversion elements within a pixel region can be arranged in one dimension. It is not always necessary to configure the signal processing unit for each and every photoelectric conversion element. For example, a signal processing unit can be shared by multiple photoelectric conversion elements, and signal processing can be performed sequentially.

[0054] like Figure 2 and Figure 3 As illustrated in the diagram, multiple signal processing units 103 are arranged in a region overlapping with the pixel region 12 in a planar view. Then, the vertical scanning circuit unit 110, the horizontal scanning circuit unit 111, the readout circuit 112, the output circuit 114, and the control pulse generation unit 115 are arranged to overlap with the region defined by the ends of the sensor substrate 11 and the ends of the pixel region 12 in the planar view. In other words, the sensor substrate 11 includes the pixel region 12 and a non-pixel region arranged around the pixel region 12. Then, the vertical scanning circuit unit 110, the horizontal scanning circuit unit 111, the readout circuit 112, the output circuit 114, and the control pulse generation unit 115 are arranged in a region overlapping with the non-pixel region in the planar view.

[0055] Figure 4 The diagram includes Figure 2 and Figure 3 An example of a block diagram of the equivalent circuit.

[0056] exist Figure 4 In this circuit, a photoelectric conversion element 102, including an APD 201, is disposed on a sensor substrate 11, and other components are disposed on a circuit substrate 21.

[0057] APD 201 is a photoelectric conversion unit that generates charge pairs corresponding to incident light through photoelectric conversion.

[0058] A voltage VL (first voltage) is supplied to the anode of the APD 201. A voltage VH (second voltage), which is higher than the voltage VL supplied to the anode, is supplied to the cathode of the APD 201. A reverse bias voltage for causing the APD 201 to perform avalanche multiplication operation is supplied to both the anode and cathode. By inducing the state of supplying this voltage, the charge generated by the incident light causes avalanche multiplication, and an avalanche current is generated.

[0059] With a reverse bias voltage supplied, the APD operates in either Geiger mode or linear mode. In Geiger mode, the APD operates with a potential difference between the anode and cathode greater than the breakdown voltage. In linear mode, the APD operates with a potential difference between the anode and cathode close to the breakdown voltage, or with a voltage difference equal to or less than the breakdown voltage.

[0060] An APD operating in Geiger mode will be called a single-photon avalanche photodiode (SPAD). For example, the voltage VL (first voltage) is -30V and the voltage VH (second voltage) is 1V. The APD 201 can operate in linear mode or in Geiger mode. Because the potential difference of a SPAD is larger and its breakdown voltage effect becomes more prominent compared to an APD in linear mode, the use of a SPAD is desirable.

[0061] Quenching element 202 is connected to APD 201 and the power supply for voltage VH. When the signal is amplified by avalanche multiplication, quenching element 202 acts as a load circuit (quenching circuit) and has the function of suppressing avalanche multiplication by reducing the voltage to be supplied to APD 201 (quenching operation). Quenching element 202 also has the function of returning the voltage to be supplied to APD 201 to voltage VH by allowing current to flow in an amount corresponding to the voltage drop caused by the quenching operation (recharging operation).

[0062] The signal processing unit 103 includes a waveform shaping unit 210, a counter circuit 211, and a selection circuit 212. In this specification, the signal processing unit 103 only needs to include any one of the waveform shaping unit 210, the counter circuit 211, and the selection circuit 212.

[0063] The waveform shaping unit 210 outputs a pulse signal by shaping the potential change of the cathode of the APD 201 obtained during photon detection. An example of the waveform shaping unit 210 includes an inverter circuit. Figure 4 The illustration shows an example of an inverter being used as waveform shaping unit 210, but a circuit in which multiple inverters are connected in series can be used, or another circuit with waveform shaping effect can be used.

[0064] The counter circuit 211 counts the number of pulse signals output from the waveform shaping unit 210 and holds the count value. When a control pulse pRES is supplied via the drive line 213, the number of pulse signals held in the counter circuit 211 is reset.

[0065] via Figure 4 The driving line 214 shown in the figure ( Figure 3 (not shown in the image), from Figure 3The vertical scanning circuit unit 110 shown in the figure supplies control pulse pSEL to the selection circuit 212, and the electrical connection and disconnection between the counter circuit 211 and the signal line 113 are selected. The selection circuit 212 includes, for example, a buffer circuit for the output signal.

[0066] Electrical connections can be switched by arranging switches, such as transistors, between quenching element 202 and APD 201 or between photoelectric conversion element 102 and signal processing unit 103. Similarly, switches, such as transistors, can be used to electrically switch the supply of voltage VH or voltage VL to be supplied to photoelectric conversion element 102.

[0067] In this embodiment, the configuration using counter circuit 211 has been described. However, instead of counter circuit 211, photoelectric conversion device 100 can use a time-to-digital converter (hereinafter, TDC) and a memory to acquire pulse detection timing. In this case, the generation timing of the pulse signal output from waveform shaping unit 210 is converted into a digital signal by TDC. To measure the timing of the pulse signal, from... Figure 3 The vertical scanning circuit unit 110 shown in the diagram supplies a control pulse pREF (reference signal) to the TDC via a drive line. Based on the control pulse pREF, the TDC acquires a signal as a digital signal that is an input timing indication of the signal output from each pixel via the waveform shaping unit 210 as a relative time.

[0068] Figure 5A , Figure 5B and Figure 5C It is a schematic diagram illustrating the relationship between the operation of the APD and its output signal.

[0069] Figure 5A It is extraction Figure 4 The diagram shows the APD 201, quenching element 202, and waveform shaping unit 210. Figure 5A In this context, the input side of the waveform shaping unit 210 is considered as node A, and the output side is considered as node B. Figure 5B The diagram shows Figure 5A The waveform change of node A in the diagram, and Figure 5C The diagram shows Figure 5A The waveform change of node B in the diagram.

[0070] During the time interval from time t0 to time t1, the potential difference VH-VL is applied to Figure 5AThe diagram shows APD 201. If a photon enters APD 201 at time t1, avalanche multiplication occurs in APD 201, the avalanche multiplication current flows to quenching element 202, and the voltage at node A drops. If the voltage drop further increases and the potential difference applied to APD 201 decreases, then at time t2, avalanche multiplication in APD 201 stops, and the voltage level at node A stops decreasing from a certain fixed value. Thereafter, during the period from time t2 to time t3, current compensating for the voltage drop from voltage VL flows to node A, and at time t3, the potential level at node A statically stabilizes at the original potential level. At this time, the portion of the output waveform at node A that exceeds a certain threshold undergoes waveform shaping performed by waveform shaping unit 210 and is output as a signal at node B.

[0071] The arrangement of signal line 113 and the arrangement of readout circuit 112 and output circuit 114 are not limited to the following. Figure 3 The arrangement is shown in the diagram. For example, signal line 113 can be arranged extending in the row direction, and readout circuitry 112 can be arranged at the end of the extended signal line 113.

[0072] The photoelectric conversion device of each embodiment will be described below.

[0073] Reference Figures 6 to 10 A processing apparatus according to a first embodiment is described.

[0074] Figure 6 This is an explanatory diagram of a comparative example of the first embodiment.

[0075] As a general defect correction unit, it is known to extract pixels whose output level is higher than that of surrounding pixels as defect pixels and replace the output of the defect pixel with the average or median value of the output of the surrounding pixels.

[0076] For example, in the case of an APD (Automatic Photodiode) pixel, multiple defective pixels may occur in some cases due to crosstalk caused by avalanche luminescence, which is attributed to the decoupling of charges subjected to avalanche multiplication. Such defects are referred to as cluster defects. The occurrence of cluster defects due to crosstalk is not limited to the case of APD pixels, and cluster defects can also occur in complementary metal-oxide-semiconductor (CMOS) sensors.

[0077] Figure 6 (a) is a schematic illustration of the cluster defects before correction. It shows that the whiter the pixel, the higher the output value. Figure 6In (a), besides the central defect pixel, the influence of the central defect pixel extends to the two pixels above and below the defect pixel, the two pixels to the left and right of the defect pixel, and the four pixels diagonally opposite the central defect pixel. A method is applied to replace the output of pixels with this cluster defect with the average output of the surrounding pixels. Figure 6 Figure (b) illustrates the correction unit. The correction unit replaces the output of the center defect pixel with the average of the outputs of the left and right pixels. However, if this correction unit is applied to... Figure 6 The example shown in (a) is as follows: Figure 6 As illustrated in (c), residual defects remain, and the correction accuracy deteriorates. Therefore, if the correction unit of the comparative example is used, there is a problem of deteriorated image quality after defect correction.

[0078] Figure 7 This is an explanatory diagram of the first embodiment. In this embodiment, we first look at the image that generates cluster defects ( Figure 7 (a)) Extract pixels with output levels higher than surrounding pixels as defect pixels. Figure 7 (b)). At this time, the zero least significant bit (0LSB) is set to the output of the surrounding pixels other than the pixel extracted as the defect pixel. By multiplying the obtained defect pixel by the pre-obtained correction array ( Figure 7 (C)), to obtain the estimated defect distribution ( Figure 7 (d)). The correction array indicates the effect of a defective pixel on its surrounding pixels as a matrix of crosstalk probabilities (crosstalk matrix). For example, multiplication includes a calculation that multiplies each component of the crosstalk matrix by a constant using the numerical values ​​based on the defective pixel output, and a convolution operation using a Fast Fourier Transform (FFT). Through this process, the defect distribution of cluster defects can be estimated from the extracted defective pixels. By performing cluster defect correction using the obtained estimated defect distribution, high-precision defect correction can be performed without leaving residual defects.

[0079] Figure 8 This is a block diagram of the processing apparatus according to the first embodiment.

[0080] The processing apparatus according to the first embodiment includes an image acquisition unit 301, a readout unit 302, and an image processing unit 303. The image processing unit 303 includes a first storage unit 304, a defect extraction unit 305, a second storage unit 306, and a correction unit 310. The correction unit 310 includes an array generation unit 311 and a calculation unit 312.

[0081] Image acquisition unit 301 corresponds to, for example Figure 2 The pixel 101 in pixel region 12 shown in the figure, and the readout unit 302 corresponds to, for example Figure 3The signal processing unit 103 shown in the figure. The image data generated by the image acquisition unit 301 is input to the first storage unit 304.

[0082] The first storage unit 304 stores at least a portion of the image data generated by the previous unit as first array data. Specifically, a memory such as static random access memory (SRAM) is used as the storage unit to store the first array data.

[0083] The defect extraction unit 305 extracts the output value of the defective pixel from the image data stored in the first storage unit 304 as the fifth array data, and identifies the position information and output value of the defective pixel.

[0084] The second storage unit 306 stores the second array data used for correction. That is, the second array data used for correction is a crosstalk matrix.

[0085] The array generation unit 311 generates third array data as correction calculation data based on the output value and position of the defective pixel extracted by the defect extraction unit 305 and the second array data for correction stored in the second storage unit 306.

[0086] The calculation unit 312 performs image data correction based on the third array data generated by the array generation unit 311.

[0087] Figure 9 This is a flowchart illustrating the process to be performed by the processing apparatus according to the first embodiment.

[0088] In step S401, the defect extraction unit 305 selects defect pixels based on defect address data from the image data acquired by the image acquisition unit 301. It is assumed that the defect address data is the pre-acquired location information of the defect pixels.

[0089] In step S402, the output value of the defective pixel selected in step S401 is obtained.

[0090] In step S403, the second storage unit 306 selects correction array data. The correction array data includes N rows and M columns. The second storage unit 306 can store various types of correction array data, which will be described in detail below.

[0091] In step S404, an estimated defect array of N×M pixels is generated.

[0092] In step S405, the process of subtracting the estimated defect array from the image data of the N×M pixel region including the defect pixels is performed.

[0093] In step S406, the output value with a negative value after the subtraction in step S405 is replaced with 0, and the process ends.

[0094] The correction array data can be one-dimensional or two-dimensional, as long as it comprises two or more data blocks. In other words, it is sufficient that one of N and M is an integer greater than or equal to 2 and the other is an integer greater than or equal to 1. Ideally, the correction array data comprises three or more data blocks, and both N and M are integers greater than or equal to 3. In other words, it is desirable that the correction array data has a size greater than or equal to three rows and three columns, and the pixel data to be corrected also has a size greater than or equal to three rows and three columns. This is because, in this embodiment, correction can be performed accurately even when pixels exist around the defective pixel. Furthermore, considering the symmetry of the crosstalk probability, it is desirable that N and M be odd numbers.

[0095] In this embodiment, pre-acquired address data is used to detect defective pixels, but the defect extraction method is not limited to this. Because cluster defects typically have cross, square, or circular shapes, the output value of defective pixels can be detected, for example, by pattern matching.

[0096] Furthermore, in the above description, correction is performed by subtracting the estimated defect array data from the image data including the defects. However, the correction process can be any other than subtraction. For example, correction can be performed by dividing the image data including the defects by the generated defect array. Alternatively, if the inverse of the probability that a defective pixel affects its surrounding pixels is used as the correction array, multiplication of the image data including the defects with the estimated defect array data can be performed. In this case, the base value is assigned to the position (center) in the correction array corresponding to the defective pixel.

[0097] Figure 10 This is an explanatory diagram of the correction array of the processing apparatus according to the first embodiment.

[0098] As mentioned above, in Figure 9 In step S403, the calibration array data is selected. The second storage unit 306 stores, for example, three types of calibration arrays corresponding to calibration arrays A to C. Figure 10 (a) is a schematic diagram corresponding to the cluster defects of the correction array A. Figure 10 (b) is a schematic diagram corresponding to the cluster defects of the correction array B, and Figure 10 (c) is a schematic diagram corresponding to the cluster defects of the correction array C.

[0099] Correction array A is a three-row, three-column correction array to be used when the defective pixel affects four pixels arranged above, below, to the left and to the right of the defective pixel.

[0100] Correction array B is a five-row, five-column correction array to be used when the defective pixel affects a total of 12 pixels. The 12 pixels include two pixels above the defective pixel, two pixels below the defective pixel, two pixels to the left of the defective pixel, two pixels to the right of the defective pixel, and four pixels arranged diagonally opposite the defective pixel.

[0101] Similarly, the correction array C comprises seven rows and seven columns, intended for use when a defective pixel affects 16 pixels. These 16 pixels include three pixels above the defective pixel, three pixels below the defective pixel, three pixels to the left of the defective pixel, three pixels to the right of the defective pixel, and four pixels arranged diagonally opposite the defective pixel. Because light, due to crosstalk, tends to travel in the opposite lateral directions of pixels, the influence of the defective pixel is considered in a wider area in both the vertical and horizontal directions compared to the diagonal direction.

[0102] In step S403, a correction array is generated by selecting any one of these correction arrays or by performing a weighted summation on these correction arrays. The correction array can be selected based on the image capture environment, the temperature measurement results of the sensor generating the image data, the voltage conditions to be applied to the image acquisition unit and processing device, and the driving conditions of the image acquisition unit and processing device. In the presence of different pixel layouts, the correction array can be selected based on the pattern of the pixel layout. Alternatively, the correction array can be selected based on the arrangement points in the pixel array of the pixel to be corrected. Specifically, when the target pixel is located at the end of the pixel array, because the number of surrounding pixels causing crosstalk is small, array data with a small size can be used.

[0103] If a row or column of a calibration array, which is considered as two-dimensional array data, is extracted as one-dimensional data, the one-dimensional data has a peak at the center. The one-dimensional data also exhibits a monotonically varying distribution from the data center corresponding to the peak towards the data ends that are the two ends of the one-dimensional data. Furthermore, one-dimensional data blocks arranged in the row and column directions, sharing data centers with peak values ​​and arranged in intersecting directions, exhibit a monotonically varying distribution in both the row and column directions from the data center towards the data ends.

[0104] For example, when two adjacent pixels (pixel A and pixel B) both output peak values, pixels A and B each have a correction array comprising a data distribution centered on the corresponding pixel. In this arrangement, due to shot noise or quantization errors, sometimes over-correction is performed on pixels near both pixels A and B. However, because... Figure 9As illustrated in step S406, pixel outputs with negative values ​​after correction processing due to excessive correction processing are replaced with 0, so the image quality is not degraded by the processing. In this embodiment, an example of replacing pixel outputs with 0 has been described. Alternatively, the pixel output can be replaced with the average or median value of the output values ​​of pixels surrounding the defective pixel, or it can be replaced with a predetermined non-negative constant.

[0105] Even when a defective pixel is surrounded by non-defective pixels, the output of surrounding pixels affected by the defective pixel varies due to shot noise, and in some cases, the output obtained after correction randomly becomes negative. Furthermore, in such cases, the pixel output obtained after correction is replaced with 0. Therefore, image quality degradation attributable to overcorrection can be prevented.

[0106] The above description mainly addresses the case where the correction array has a peak value at the center. Furthermore, when the correction array has a low value at the center, it similarly exhibits a monotonically varying distribution from the data center towards the data edge.

[0107] For example, a correction array can be generated based on a pattern of defective pixels in an optically black (OB) region that is shielded by a light-shielding film on the light-incident surface side of pixel region 12, which serves as the image acquisition unit. Alternatively, the correction array can be generated using acquired image data blocks through pattern learning based on machine learning or deep learning.

[0108] Now, refer to Figure 11 and Figure 12 The processing apparatus according to the second embodiment will be described. The differences from the first embodiment will be mainly described, and descriptions of parts similar to those in the first embodiment will be omitted.

[0109] Figure 11 This is a block diagram illustrating a processing apparatus according to a second embodiment.

[0110] The processing apparatus according to the second embodiment includes an image acquisition unit 301, a readout unit 302, and an image processing unit 303. The image processing unit 303 includes a gain adjustment unit 313 and a de-mosaic unit 315. The correction unit 310 includes a nonlinear correction unit 314.

[0111] Gain adjustment unit 313 is disposed between image acquisition unit 301 (readout unit 302) and first storage unit 304, and applies digital gain to the image data generated by image acquisition unit 301. Correction array data, based on crosstalk probability and stored in second storage unit 306, typically includes fractional values. If the array generated by array generation unit 311 is integer-type, and the image output is integer-type, the correction accuracy may be degraded due to quantization error. By pre-applying gain to the image data, the effects of quantization error can be suppressed and correction accuracy improved. If the quantization error can be reduced to 1 / 4 or less of the single-photon signal level, the corrected image becomes visually natural. Therefore, for example, it is desirable to apply four or more digital gains to the image data.

[0112] A nonlinear correction unit 314 is arranged between the first storage unit 304 and the computation unit 312, and corrects the image data. In the case where the image acquisition unit 301 is a photon counting detector, the optical response often becomes nonlinear due to the effect of the dwell time. Emission crosstalk increases the signal level when superimposed on the light signal to be received by surrounding pixels, but if correction is performed under the assumption of a nonlinear optical response, overcorrection can sometimes occur when the nonlinear optical response is received. Therefore, overcorrection can be prevented by performing nonlinear correction of the image data before the computational processing performed by the computation unit 312.

[0113] For example, lookup tables can be used to perform nonlinear corrections.

[0114] The demosaic unit 315 is configured as a subsequent unit to the computation unit 312. When the image acquisition unit 301 is a device capable of acquiring a color image, it is necessary to perform demosaic processing to supplement the signal acquired by the image acquisition unit 301 with color information from neighboring pixels. Crosstalk occurs regardless of the pixel color array, therefore it is desirable to perform crosstalk correction before the demosaic processing. When the image acquisition unit 301 is a device capable of acquiring a color image, it is desirable for the defect extraction unit 305 to perform defect determination by comparing the output of the pixel of interest with the output of neighboring pixels having the same color as the pixel of interest.

[0115] Figure 12 This is a flowchart illustrating the process to be performed by the processing apparatus according to the second embodiment.

[0116] In step S501, pixels are selected from the image data acquired by the image acquisition unit 301.

[0117] In step S502, defective pixel determination is performed. In this step, if the difference between the output value of the selected pixel and its surrounding pixels is greater than or equal to a threshold ("Yes" in step S502), the selected pixel is determined to be a defective pixel. If the selected pixel is not a defective pixel ("No" in step S502), the process returns to step S501. In step S501, a pixel is selected again, and the processes in steps S501 and S502 are repeated until the selected pixel is determined to be a defective pixel. If the difference between the output value of the selected pixel and its surrounding pixels is greater than or equal to a threshold ("Yes" in step S502), the selected pixel is determined to be a defective pixel, and the process proceeds to step S503. In step S503, correction array data is selected.

[0118] In step S504, an estimated defect array of N×M pixels is generated.

[0119] In step S505, the estimated defect array is added to the N×M pixel region of the subtraction image.

[0120] In step S506, the subtracted image is removed from the image data.

[0121] In step S507, the output value that has a negative value after the subtraction in step S506 is replaced with 0.

[0122] In step S508, intermediate value correction is performed on the pixels that were identified as defective pixels in step S502, and the process ends.

[0123] In many cases, the spurious signal level in the surrounding pixels of a defective pixel is relatively low within the cluster defect, allowing for accurate defect correction through subtraction. In contrast, defective pixels located at the center of the cluster defect have relatively high outputs, potentially leading to errors during correction due to shot noise. Correction accuracy can be improved by initially correcting the outputs of surrounding pixels through subtraction, and then correcting the output of the defective pixel based on the corrected outputs of the surrounding pixels.

[0124] In this embodiment, the correction accuracy as a whole image is improved by correcting the output of the surrounding pixels of the defective pixel through subtraction, and then replacing the output of the defective pixel with the average or median value of the corrected output of the surrounding pixels. The offset output based on the OB region can be added to the subtracted image, or the offset of the dark current component can be subtracted by reflecting the output distribution that takes into account the occlusion of the dark output.

[0125] In this embodiment, a selected pixel is identified as a defective pixel if the difference between its output value and that of its neighboring pixels is greater than or equal to a threshold. For example, neighboring pixels refer to pixels adjacent to the selected pixel. In other words, either the first pixel or the second pixel is identified as a defective pixel if the output difference between its output value and that of a second pixel adjacent to it is greater than or equal to a fixed threshold. In this case, defect correction can be performed without storing defect address data, thus reducing storage capacity and system size.

[0126] In this embodiment, correction is performed by subtracting a subtracted image generated by adding estimated defect array data from the image data including defects, rather than directly subtracting the estimated defect array data from the image data including defects. When defect correction is performed pixel-by-pixel, pixels affected by crosstalk from multiple defective pixels undergo multiple subtraction processes. Depending on the method used in the correction process, quantization error is generated each time subtraction is performed, and the defect correction accuracy deteriorates. As described in this embodiment, by sequentially adding the estimated defect array used for correction to the subtracted image, quantization error can be reduced and the accuracy of defect correction can be improved. As mentioned above, the correction of the output of pixels surrounding the defective pixel can be performed using a correction process that employs calculations such as division or multiplication other than subtraction. In this case, the degradation of correction accuracy caused by quantization error can also be prevented.

[0127] Figure 13 This is a flowchart illustrating the process to be performed by the processing apparatus according to the third embodiment. In this embodiment, whether to perform defect correction may be switched depending on the image capture scenario.

[0128] In step S601, the average signal level of the image data acquired by the image acquisition unit is compared with a threshold. If the average signal level is not less than or equal to the threshold ("No" in step S601), the process proceeds to step S609. In step S609, the defect correction process is stopped.

[0129] This is because, in scenarios where the average signal level is not less than or equal to the threshold (i.e., the signal level is high), the skirt portion of cluster defects is not very noticeable and can be adequately corrected using standard defect correction algorithms.

[0130] If the average signal level is less than or equal to a threshold (i.e., in a low-light environment) (in step S601, "Yes"), cluster defect correction is expected to be performed. In this case, the process proceeds to step S602, which involves selecting pixels from the acquired image.

[0131] In step S603, defective pixel determination is performed. If the difference between the output value of the selected pixel and its surrounding pixels is greater than or equal to a threshold ("Yes" in step S603), the selected pixel is determined to be a defective pixel. If the difference between the output value of the selected pixel and its surrounding pixels is not greater than or equal to the threshold ("No" in step S603), the process returns to step S602. In step S602, a pixel is selected again, and the operations in steps S602 and S603 are repeated until the selected pixel is determined to be a defective pixel. If the ratio between the obtained pixel and the output value of its surrounding pixels is greater than or equal to a threshold ("Yes" in step S603), the process proceeds to step S604. In step S604, correction array data is selected.

[0132] In step S605, an estimated defect array of N×M pixels is generated.

[0133] In step S606, the estimated defect array is added to the region corresponding to N×M pixels in the subtracted image.

[0134] The subtracted image can be generated by repeating the operations in steps S602 to S606.

[0135] In step S607, the subtracted image is removed from the image data.

[0136] In step S608, the output value with a negative value after subtraction is replaced with 0, and the process ends.

[0137] Based on the process described in this embodiment, typical defect correction is performed in high-illuminance environments where the skirt portion of cluster defects is not very noticeable, and cluster defect correction is performed in low-illuminance environments where cluster defects are obvious. By switching whether to perform defect correction depending on the image capture scene in this way, both high-precision defect correction and reduced processing time can be achieved.

[0138] Figure 14 This is a flowchart illustrating the process to be performed by the processing apparatus according to the fourth embodiment. In this embodiment, by updating the defect information for each frame group comprising multiple frames, more accurate defect correction is made possible.

[0139] In step S701, a new frame image is acquired by the image acquisition unit.

[0140] In step S702, the stored subtracted image is subtracted from the image obtained in step S701.

[0141] In step S703, the output value with a negative value after subtraction is replaced with 0, and the result is output to the subsequent unit. The cluster defect correction processing of the new frame image obtained in step S701 is completed by the processing performed in the steps preceding step S703. The subsequent processing is the subtraction image update processing used to perform cluster defect correction processing on the next frame image.

[0142] In step S704, pixels are selected from the image data obtained before the subtraction was performed in step S703.

[0143] In step S705, defective pixel determination is performed. If the output value of the selected pixel is greater than or equal to the threshold ("Yes" in step S705), the selected pixel is determined to be a defective pixel. If the output value of the selected pixel is not greater than or equal to the threshold ("No" in step S705), the process returns to step S704. In step S704, a pixel is selected again, and the operations in steps S704 and S705 are repeated until the selected pixel is determined to be a defective pixel. If the output value of the selected pixel is greater than or equal to the threshold ("Yes" in step S705), the process proceeds to step S706. In step S706, correction array data is selected.

[0144] In step S707, an estimated defect array of N×M pixels is generated.

[0145] In step S708, the estimated defect array is added to the region corresponding to N×M pixels in the subtracted image.

[0146] The subtracted image can be generated by repeating the operations in steps S704 to S708.

[0147] In step S709, the subtracted image is stored.

[0148] According to this embodiment, for example, the correction process can be accelerated by using the defect address data obtained in a certain frame for the correction of the image data to be obtained in the next frame. The acceleration of the correction process can also be achieved by extracting the defect address once every multiple frames and using the obtained defect address data for the correction of the image data obtained in each frame until the next extraction of the defect address.

[0149] In other words, correction of image data acquired later than the first frame can be performed based on the output values ​​or address data of defective pixels extracted from the image data in the first frame.

[0150] The following will refer to Figures 15 to 17 The fifth embodiment will now be described.

[0151] In this embodiment, regarding the image that generates cluster defects ( Figure 15 (a) uses a pre-acquired correction array to perform phase division. Figure 15 (b)). Division is, for example, a deconvolution operation. Defect correction is performed on the pixels obtained after the calculation. Figure 15 (d)

[0152] exist Figure 7 In this approach, defect pixels are extracted from the image containing cluster defects. The extracted defect pixels are then multiplied by a correction array to obtain an estimated defect distribution. This estimated defect distribution is then used to correct the image containing the cluster defects. In contrast, in... Figure 15 In this embodiment illustrated in the diagram, multiple pixels, including defective pixels, are divided by the correction array without extracting the defective pixels. By performing such a calculation, defective pixels that have caused cluster defects can be effectively extracted. (Compared to reference...) Figure 7 Compared to the described embodiments, this embodiment involves fewer computational processes, thus enabling high-speed and high-precision defect correction. Furthermore, in this embodiment, defect correction can be performed accurately even when multiple defects exist in close proximity and the skirt portions overlap.

[0153] The defect correction performed on the calculated image is to replace the output value of the defective pixel with the average of the output values ​​of the surrounding pixels. For example, performing a reference... Figure 6 The defect correction method is described. The correction array indicates the effect of a defective pixel on its surrounding pixels as a matrix of crosstalk probabilities (crosstalk matrix).

[0154] Figure 16 This is a block diagram illustrating a processing apparatus according to a fifth embodiment.

[0155] The processing apparatus according to the fifth embodiment includes an image acquisition unit 301, a readout unit 302, and an image processing unit 303. The image processing unit 303 includes a first storage unit 304, a second storage unit 306, and a correction unit 310. The correction unit 310 includes a calculation unit 312 and a defect correction unit 320.

[0156] Image acquisition unit 301 corresponds to, for example Figure 2 The pixel 101 in pixel region 12 shown in the figure, and the readout unit 302 corresponds to, for example Figure 3 The signal processing unit 103 shown in the figure. The image data generated by the image acquisition unit 301 is input to the first storage unit 304.

[0157] The first storage unit 304 is used to store at least a portion of the image data generated by the previous unit as first array data. Specifically, the first array data is stored in a memory such as SRAM, which serves as the storage unit.

[0158] The second storage unit 306 stores the second array data used for correction. That is, the second array data used for correction is a crosstalk matrix. The second storage unit 306 can store various types of correction array data.

[0159] The calculation unit 312 performs calculations on the first array data stored in the first storage unit 304 based on the second array data stored in the second storage unit 306, and obtains the calculated image as the fourth array data. The calculation unit 312 can perform deconvolution operations on the crosstalk matrix stored in the second storage unit 306, or it can perform convolution operations on the matrix (point spread function) corresponding to the inverse transformation of the crosstalk matrix stored in the second storage unit 306. Through the calculations performed by the calculation unit 312, cluster defects in the image data can be corrected to a state close to point defects.

[0160] The defect correction unit 320 is a correction processing unit that performs defect correction processing on the corrected image obtained by the calculation unit 312. The defect correction unit 320 functions as a defect extraction unit that extracts defective pixels, and performs defect correction processing on the output of the extracted defective pixels.

[0161] Figure 17 This is a flowchart illustrating the process to be performed by the processing apparatus according to the fifth embodiment.

[0162] In step S801, the calibration array data is selected.

[0163] In step S802, the image data undergoes a deconvolution operation on the correction array data selected in step S801.

[0164] In step S803, defects are extracted from the image obtained after the deconvolution operation in step S802. Defect extraction refers to the identification of the location information or output value of defective pixels in the image.

[0165] In step S804, defects in the image obtained after the deconvolution operation are corrected. For example, the output value of the defective pixel extracted in step S803 is replaced with the average of the output values ​​of the surrounding pixels.

[0166] According to the processing apparatus of this embodiment, it is possible to correct cluster defects using a defect correction algorithm for point defects.

[0167] Now, refer to Figure 18 and Figure 19The sixth embodiment will now be described. The main differences from the fifth embodiment will be described, and descriptions of parts similar to those in the fifth embodiment will be omitted.

[0168] Figure 18 This is a block diagram of a processing apparatus according to the sixth embodiment.

[0169] The processing apparatus according to the sixth embodiment includes an image acquisition unit 301, a readout unit 302, and an image processing unit 303. The correction unit 310 includes a first calculation unit 312A, a second calculation unit 312B, and a defect correction unit 320.

[0170] The defect correction unit 320 is disposed between the first calculation unit 312A and the second calculation unit 312B. A feature of this embodiment is that it performs both convolution and deconvolution operations on the crosstalk matrix of the image data.

[0171] Figure 19 This is a flowchart illustrating the process to be performed by the processing apparatus according to the sixth embodiment.

[0172] In step S901, the calibration array data is selected.

[0173] In step S902, the image data undergoes a deconvolution operation on the correction array data selected in step S901.

[0174] In step S903, defects are extracted from the image obtained after the deconvolution operation.

[0175] In step S904, the defects extracted from the image obtained after the deconvolution operation in step S903 are corrected.

[0176] In step S905, a convolution operation of the correction array data is performed on the defect-corrected image.

[0177] In the fifth embodiment, random noise in the region other than the defective pixels to be corrected is amplified by the deconvolution operation of the crosstalk matrix, and in some cases, image quality deteriorates. According to this embodiment, image quality degradation can be compensated by convolving the crosstalk matrix into the defect-corrected image data. Therefore, both improved defect correction accuracy and suppression of image quality degradation attributable to signal processing can be achieved. Similar to the fifth embodiment, the second storage unit 306 can store the crosstalk matrix, and can also store the matrix corresponding to the inverse transformation of the crosstalk matrix.

[0178] Now, refer to Figures 20 to 22 The seventh embodiment will now be described. Descriptions of parts similar to those in the first through sixth embodiments will be omitted.

[0179] In this embodiment, regarding the image that generates cluster defects ( Figure 20 (a)) In performing nonlinear correction ( Figure 20 (b) At the same time, by multiplying the image by the correction array ( Figure 20 The estimated distribution of the output generated by crosstalk is obtained by (C). Furthermore, cluster defects are corrected by subtracting the estimated distribution image from the nonlinearly corrected image and then performing point defect correction processing.

[0180] When the technology of the present invention is applied to an APD including an active recharge type drive circuit represented by a clock recharge type, the number of counts decreases relative to the number of incident photons due to the accumulation phenomenon, and the response characteristics of photoelectric conversion sometimes exhibit nonlinearity.

[0181] When the nonlinear response function is expressed by the following equation:

[0182]

[0183] Furthermore, when the number of counts is represented by Nct and the number of incident photons by Nph, we obtain Nct = f(Nph). In this case, the amount of light emitted due to crosstalk is not proportional to the number of incident photons Nph, but rather to the number of counts Nct. On the other hand, the total number of photons entering the crosstalk-affected pixel, Nct', becomes Nc + Ns, where Nc is the pseudo-signal attributable to crosstalk, and Ns is the number of signal photons. In other words, we obtain Nct' = f(Nc + Ns). Therefore, when estimating the distribution of pseudo-signal photons generated by crosstalk, if the estimation is performed based on the number of counts obtained after nonlinear correction, the difference from the actual distribution of pseudo-signal photons may become significant.

[0184] In light of the above, it is appropriate to estimate the distribution of spurious signal photons generated by crosstalk based on the number of counts obtained before performing nonlinear correction. By subtracting the obtained distribution from the nonlinearly corrected image data, correction errors due to cluster defects can also be suppressed in scenes including high-output pixels that cause nonlinear effects.

[0185] In this embodiment, by setting 0 to the value at the center of the correction array, overcorrection of the center pixel output can be prevented by subtracting only the crosstalk effect of the center pixel of the cluster defect on the surrounding pixels. Furthermore, by subtracting the estimated output distribution caused by crosstalk from the nonlinearly corrected image, correction can be performed accurately even when cluster defects are present in proximity or even when the signal light intensity corresponding to the target subject in the image capture is high.

[0186] Figure 21 This is a block diagram of a processing apparatus according to the seventh embodiment.

[0187] The processing apparatus according to the seventh embodiment includes an image acquisition unit 301, a readout unit 302, and an image processing unit 303. The image processing unit 303 includes a gain adjustment unit 313, a first storage unit 304, a second storage unit 306, a correction unit 310, and a de-mosaic unit 315. The correction unit 310 includes a nonlinear correction unit 314, a calculation unit 312, an array generation unit 311, and a defect correction unit 320.

[0188] Image acquisition unit 301, for example, corresponds to Figure 2 The pixel 101 in pixel region 12 shown in the figure, and the readout unit 302 corresponds, for example, to pixel 101 in pixel region 12. Figure 3 The signal processing unit 103 shown in the figure. The image data generated by the image acquisition unit 301 is input to the first storage unit 304.

[0189] At least a portion of the image data generated in the previous unit is gain-adjusted by the gain adjustment unit 313, and the first storage unit 304 stores the gain-adjusted data as first array data. Specifically, a memory such as SRAM is used as the storage unit to store the first array data.

[0190] The second storage unit 306 stores the second array data used for correction. The second array data used for correction is a crosstalk matrix. The second storage unit 306 can store various types of correction array data.

[0191] A nonlinear correction unit 314 is arranged between the first storage unit 304 and the calculation unit 312, and corrects the image data. An array generation unit 311 generates third array data by performing calculations based on first array data stored in the first storage unit 304 and second array data stored in the second storage unit 306. For example, a convolution operation is performed. The calculation unit 312 subtracts the third array data generated by the array generation unit 311 from the array data corrected by the nonlinear correction unit 314.

[0192] The defect correction unit 320 is a correction processing unit that performs defect correction processing on the corrected image obtained by the calculation unit 312. The defect correction unit 320 functions as a defect extraction unit that extracts defective pixels, and performs defect correction processing on the output of the extracted defective pixels.

[0193] The demosaic unit 315 is set as the successor unit to the computation unit 312 and performs demosaic processing on the defect-corrected array data.

[0194] Figure 22 This is a flowchart illustrating the process to be performed by the processing apparatus according to the seventh embodiment.

[0195] In step S1001, the calibration array data is selected.

[0196] In step S1002, nonlinear correction of the acquired image is performed.

[0197] In step S1003, a convolution operation is performed on the acquired image and the correction array data.

[0198] In step S1004, the convolutional image is subtracted from the nonlinearly corrected image.

[0199] In step S1005, defect correction processing is performed on the image obtained after subtraction.

[0200] According to the processing apparatus of this embodiment, while reducing nonlinear effects, it is possible to perform cluster defect correction using a defect correction algorithm for point defects.

[0201] Now, refer to Figure 23 The photoelectric conversion system according to the eighth embodiment will be described. Figure 23 This is a block diagram illustrating a schematic configuration of the photoelectric conversion system according to this embodiment.

[0202] The processing apparatus described in the first to seventh embodiments above can be applied to various photoelectric conversion systems. Examples of photoelectric conversion systems to which the processing apparatus can be applied include digital still cameras, digital portable camcorders, surveillance cameras, copiers, fax machines, mobile phones, vehicle-mounted cameras, and observation satellites. The photoelectric conversion systems described above also include camera modules comprising optical systems such as lenses and imaging devices. As examples of these photoelectric conversion systems, Figure 23 An exemplary block diagram of a digital still camera is shown.

[0203] Figure 23 The illustrated photoelectric conversion system includes an imaging device 1004, serving as an example of a photoelectric conversion apparatus, and a lens 1002 that forms an optical image of a subject on the imaging device 1004. The photoelectric conversion system also includes an aperture 1003 for varying the amount of light passing through the lens 1002 and a baffle 1001 for protecting the lens 1002. The lens 1002 and aperture 1003 serve as an optical system for focusing light onto the imaging device 1004. The imaging device 1004 is a photoelectric conversion apparatus according to any of the above embodiments and converts the optical image formed by the lens 1002 into an electrical signal.

[0204] The photoelectric conversion system also includes a signal processing unit 1007, which serves as an image generation unit. The signal processing unit 1007 generates an image by processing the output signal from the imaging device 1004. The signal processing unit 1007 performs operations on the output image data as needed, after performing various types of corrections and compressions. The processing apparatus according to the first to fourth embodiments can be formed as part of the signal processing unit 1007. The signal processing unit 1007 can be formed on a semiconductor substrate on which the imaging device 1004 is disposed, or it can be formed on a semiconductor substrate different from the semiconductor substrate of the imaging device 1004.

[0205] The photoelectric conversion system also includes a memory unit 1010 for temporarily storing image data and an external interface unit (external I / F unit) 1013 for communicating with an external computer. The photoelectric conversion system also includes a recording medium 1012, such as a semiconductor memory, for recording or reading captured image data, and a recording medium control I / F unit 1011 for performing readings onto or from the recording medium 1012. The recording medium 1012 may be built into the photoelectric conversion system or may be detachably attached to the photoelectric conversion system.

[0206] The photoelectric conversion system also includes an overall control / computing unit 1009 for controlling various types of calculations and the entire digital still camera, and a timing signal generation unit 1008 for outputting various timing signals to the imaging device 1004 and the signal processing unit 1007. The timing signals can be input externally. The photoelectric conversion system only needs to include at least the imaging device 1004 and the signal processing unit 1007 for processing the output signals from the imaging device 1004.

[0207] Imaging device 1004 outputs an imaging signal to signal processing unit 1007. Signal processing unit 1007 performs predetermined signal processing on the imaging signal output from imaging device 1004 and then outputs image data. Signal processing unit 1007 uses the imaging signal to generate an image.

[0208] In this way, according to this embodiment, a photoelectric conversion system applying any of the processing devices described in the above embodiments can be realized.

[0209] Below, we will refer to Figure 24A and Figure 24B The photoelectric conversion system and the mobile body according to the ninth embodiment will be described. Figure 24A and Figure 24B This is a diagram illustrating the configuration of the photoelectric conversion system and the moving body according to this embodiment.

[0210] Figure 24AAn example of a photoelectric conversion system associated with a vehicle-mounted camera is illustrated. The photoelectric conversion system 2300 includes an imaging device 2310. The imaging device 2310 is a photoelectric conversion device according to any of the above embodiments. The photoelectric conversion system 2300 includes an image processing unit 2312 that performs image processing on a plurality of image data blocks acquired by the imaging device 2310. The photoelectric conversion system 2300 also includes a disparity acquisition unit 2314 that calculates disparity (phase difference between disparity images) from the plurality of image data blocks acquired by the photoelectric conversion system 2300. The processing apparatus according to the first to fourth embodiments can be formed as part of the image processing unit 2312. The photoelectric conversion system 2300 also includes: a distance acquisition unit 2316 that calculates the distance to a target object based on the calculated disparity; and a collision determination unit 2318 that determines whether a collision is possible based on the calculated distance. In this example, the disparity acquisition unit 2314 and the distance acquisition unit 2316 serve as an example of a distance information acquisition unit that acquires distance information regarding the distance to a target object. More specifically, the distance information is information about parallax, defocus, and distance to the target object. The collision determination unit 2318 can use any of this distance information to determine the probability of a collision. The distance information acquisition unit can be implemented by specially designed hardware or by a software module. Alternatively, the distance information acquisition unit can be implemented by a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC), or a combination thereof.

[0211] The photoelectric conversion system 2300 is connected to the vehicle information acquisition device 2320 and can acquire vehicle information such as vehicle speed, yaw rate, or rudder angle. Additionally, a control electronic control unit (ECU) 2330 is connected to the photoelectric conversion system 2300. The ECU 2330 acts as a control unit that outputs a control signal to apply braking force to the vehicle based on the determination results obtained by the collision determination unit 2318. The photoelectric conversion system 2300 is also connected to an alarm device 2340, which issues an alarm to the driver based on the determination results obtained by the collision determination unit 2318. For example, if the determination results obtained by the collision determination unit 2318 indicate a high probability of collision, the control ECU 2330 performs vehicle control to avoid a collision or reduce damage by braking, releasing the accelerator pedal, or suppressing engine output. The alarm device 2340 alerts the user by emitting alarms such as warning sounds, displaying warning information on the screen of the vehicle navigation system, or vibrating the seatbelt or steering wheel.

[0212] In this embodiment, the photoelectric conversion system 2300 captures images of the vehicle's periphery, such as the front or rear side. Figure 24BThe illustration shows a photoelectric conversion system 2300 for capturing an image (imaging range 2350) of the front side of a vehicle. This configuration can further enhance the accuracy of distance measurement.

[0213] The above description provides an example of control performed in a manner that avoids collisions with other vehicles. Photoelectric conversion systems can also be applied to controls for performing automated operations by following another vehicle or for performing automated operations without deviating from the driving lane. Furthermore, in addition to vehicles such as automobiles, photoelectric conversion systems can be applied to mobile bodies (mobile devices) such as ships, aircraft, or industrial robots. Moreover, besides mobile bodies, photoelectric conversion systems can also be applied to devices that widely utilize object recognition, such as Intelligent Transportation Systems (ITS).

[0214] Now, refer to Figure 25 The photoelectric conversion system according to the tenth embodiment will be described. Figure 25 This is a block diagram illustrating an example configuration of a distance image sensor used in a photoelectric conversion system according to this embodiment.

[0215] like Figure 25 As illustrated, the distance image sensor 401 includes an optical system 402, a photoelectric conversion device 403, an image processing circuit 404, a monitor 405, and a memory 406. The distance image sensor 401 can then acquire a distance image corresponding to the distance to the subject 400 by receiving light (modulated light or pulsed light) projected from the light source device 411 toward the subject 400 and reflected on the front surface of the subject 400.

[0216] The optical system 402 includes one or more lenses and forms an image on the light-receiving surface (sensor portion) of the photoelectric conversion device 403 by guiding image light (incident light) from the subject 400 to the photoelectric conversion device 403.

[0217] According to any of the above embodiments, a photoelectric conversion device is applied to the photoelectric conversion device 403, and a distance signal indicating the distance obtained from the light receiving signal output from the photoelectric conversion device 403 is supplied to the image processing circuit 404.

[0218] The image processing circuit 404 performs image processing to construct a distance image based on the distance signal supplied from the photoelectric conversion device 403. The distance image (image data) obtained through image processing is then supplied to the monitor 405 and displayed thereon, or supplied to the memory 406 and stored (recorded) therein. The processing apparatus according to the first to fourth embodiments can be formed as part of the image processing circuit 404.

[0219] By applying the photoelectric conversion device described above, the distance image sensor 401 with the above configuration can acquire a more accurate distance image, for example, by enhancing the characteristics of the pixels.

[0220] Reference Figure 26 The photoelectric conversion system according to the eleventh embodiment will be described. Figure 26 This is a diagram illustrating an example of a schematic configuration of the endoscope operating system used as the photoelectric conversion system of this embodiment.

[0221] Figure 26 The illustration shows an operator (doctor) 1131 using an endoscope operating system 1150 to perform an operation on a patient 1132 lying on a patient bed 1133. Figure 26 As shown in the diagram, the endoscope operating system 1150 includes an endoscope 1100, surgical instruments 1110, and a trolley 1134 equipped with various devices for endoscopic operation.

[0222] Endoscope 1100 includes: a lens tube 1101 having a region of predetermined length from its distal end to be inserted into a body cavity of patient 1132; and a camera head 1102 connected to the bottom end of the lens tube 1101. Figure 26 The example shown in the figure illustrates an endoscope 1100 that is formed as a so-called rigid mirror including a rigid lens barrel 1101, but the endoscope 1100 can be formed as a so-called flexible mirror including a flexible lens barrel.

[0223] An opening for embedding an objective lens is provided at the distal end of the lens barrel 1101. A light source device 1203 is connected to the endoscope 1100, and the light generated by the light source device 1203 is guided to the distal end of the lens barrel 1101 by a light guide extending inside the lens barrel 1101, and then emitted through the objective lens to the observation target in the body cavity of the patient 1132. The endoscope 1100 can be a direct-viewing endoscope, an oblique-viewing endoscope, or a lateral-viewing endoscope.

[0224] An optical system and a photoelectric conversion device are disposed inside the camera head 1102. Reflected light from the observed target (observation light) is converged by the optical system to the photoelectric conversion device. The observation light is photoelectrically converted by the photoelectric conversion device, generating an electrical signal corresponding to the observation light (i.e., an image signal corresponding to the observed image). The photoelectric conversion device according to any of the above embodiments can be used as a photoelectric conversion device. The image signal is transmitted to the camera control unit (CCU) 1135 as RAW data.

[0225] The CCU 1135 includes a central processing unit (CPU) or a graphics processing unit (GPU) and provides comprehensive control over the operation of the endoscope 1100 and the display device 1136. The CCU 1135 receives image signals from the camera head 1102 and performs various types of image processing on the image signals, such as developing (de-mosaicing), for displaying images based on the image signals.

[0226] Based on the control from CCU 1135, display device 1136 displays an image based on an image signal that has undergone image processing by CCU 1135.

[0227] The light source device 1203 includes a light source such as a light-emitting diode (LED) and supplies illumination light to the endoscope 1100 for capturing images of the operated site.

[0228] Input device 1137 is an input interface for endoscope operating system 1150. Users can input various types of information and commands into endoscope operating system 1150 via input device 1137.

[0229] The processing tool control device 1138 controls the drive of the energy processing tool 1112 to burn or cut tissue or seal blood vessels.

[0230] The light source device 1203 that supplies illumination light to the endoscope 1100 for capturing images of the surgical site can, for example, include an LED, a laser light source, or a white light source comprising a combination of these. In the case where the white light source comprises a combination of RGB laser light sources, the white balance of the captured image can be adjusted in the light source device 1203 because the output intensity and timing of each color (each wavelength) can be controlled with high precision. In this case, by emitting lasers from each RGB laser source onto the observation target in a time-division manner, and controlling the drive of the image sensor of the camera head 1102 in sync with the emission timing, images corresponding to each of the RGB values ​​can be captured in a time-division manner. According to this method, color images can be obtained without setting a color filter in the image sensor.

[0231] The drive of the light source device 1203 can be controlled to change the intensity of the light to be output at predetermined intervals. By controlling the drive of the image sensor of the camera head 1102 in time-division multiplexing in sync with the changes in light intensity, images can be acquired in time-division multiplexing, and by combining the images, high dynamic range images without underexposure of blacks and overexposure of whites can be generated.

[0232] The light source device 1203 can be configured to supply light in a predetermined wavelength band suitable for special light observation. For example, in special light observation, the wavelength dependence of light absorption in body tissues is utilized. Specifically, by emitting light in a narrower wavelength band compared to illumination light in normal observation (i.e., white light), images of predetermined tissues, such as blood vessels in the surface layer of mucous membranes, are captured with high contrast.

[0233] Alternatively, in special light observation, fluorescence observation can be performed to obtain an image by fluorescence generated by emitting excitation light. In fluorescence observation, fluorescence from body tissue can be observed by emitting excitation light onto body tissue, or a fluorescence image can be obtained by locally injecting a reagent such as indocyanine green (ICG) into body tissue and emitting excitation light of a fluorescence wavelength suitable for that reagent onto the body tissue. The light source device 1203 can be configured to supply narrowband light and / or excitation light suitable for such special light observation.

[0234] Below, we will refer to Figure 27A and Figure 27B The photoelectric conversion system according to the twelfth embodiment will be described. Figure 27A The illustration shows glasses 1600 (smart glasses) used as a photoelectric conversion system according to this embodiment. Glasses 1600 includes a photoelectric conversion device 1602. The photoelectric conversion device 1602 is the photoelectric conversion device described in any of the above embodiments. A display device including a light-emitting device such as an organic light-emitting diode (OLED) or an LED can be disposed on the back side of the lens 1601. The number of photoelectric conversion devices 1602 can be one or more. Various types of photoelectric conversion devices can be used in combination. The arrangement position of the photoelectric conversion devices 1602 is not limited to... Figure 27A The location shown in the diagram.

[0235] The eyeglasses 1600 also include a control device 1603. The control device 1603 serves as a power source for supplying power to the photoelectric conversion device 1602 and the aforementioned display device. The control device 1603 controls the operation of the photoelectric conversion device 1602 and the display device. An optical system for converging light onto the photoelectric conversion device 1602 is formed in the lens 1601.

[0236] Figure 27BThe illustration shows glasses 1610 (smart glasses) according to an application example. Glasses 1610 includes a control device 1612, which is equipped with a photoelectric conversion device equivalent to photoelectric conversion device 1602, and a display device. An optical system is formed in lens 1611 for projecting light emitted from the photoelectric conversion device and display device in the control device 1612, and an image is projected onto lens 1611. The control device 1612 serves as a power source for supplying power to the photoelectric conversion device and display device, and controls the operation of the photoelectric conversion device and display device. The control device may include a gaze detection unit for detecting the wearer's gaze. Infrared light can be used to detect the gaze. An infrared light emitting unit emits infrared light onto the eyeball of a user gazing at the displayed image. An imaging unit including a light receiving element detects the reflected light of the emitted infrared light that has been reflected from the eyeball. A captured image of the eyeball is thus obtained. Image quality degradation is suppressed by including a light reduction unit for reducing the light from the infrared light emitting unit to the display unit in a planar view.

[0237] The user's gaze relative to the displayed image is detected from a captured image of the eye obtained by image capture using infrared light. Any known method can be applied to gaze detection using a captured image of the eye. As an example, a gaze detection method based on a Purkinje image obtained by the reflection of illumination light on the cornea can be used.

[0238] More specifically, gaze detection processing based on central pupillary corneal reflection is performed. Using central pupillary corneal reflection, the user's gaze is detected by calculating an eye vector representing the direction (rotation angle) of the eyeball based on the pupil image and Purkinje image included in the captured image of the eyeball.

[0239] The display device according to this embodiment may include a photoelectric conversion device containing a light receiving element, and may control the displayed image on the display device based on the user's gaze information from the photoelectric conversion device.

[0240] Specifically, in a display device, a first field of view (FOV) and a second field of view (BAV) are determined based on gaze information. The FAV and BAV can be determined by the display device's control unit, or they can be determined by an external control unit. Within the display area of ​​the display device, the display resolution of the FAV can be controlled to be higher than that of the BAV. In other words, the resolution of the BAV can be lower than that of the FAV.

[0241] The display area includes a first display area and a second display area different from the first display area. Based on gaze information, a high-priority area can be determined from the first and second display areas. The first and second display areas can be determined by the control device of the display device, or they can be received from an external control device. The resolution of the high-priority area can be controlled to be higher than the resolution of other areas. In other words, the resolution of the relatively low-priority area can be set to a low resolution.

[0242] Artificial intelligence (AI) can be used to determine the primary visual field and areas of high priority. AI can be a model configured to estimate the angle of gaze and the distance to a distant target within the gaze from an image of the eye, using training data including images of the eye and the direction the eye is actually looking in the image. The AI ​​program can be included in a display device, a photoelectric conversion device, or an external device. In the case where the AI ​​program is included in an external device, the AI ​​program is transmitted to the display device via communication.

[0243] In cases where display control is performed based on visual detection, the present invention can be preferably applied to smart glasses that also include a photoelectric conversion device for capturing external images. The smart glasses can display external information obtained through image capture in real time.

[0244] [Modified Embodiments]

[0245] The present invention is not limited to the above embodiments, and various modifications can be made.

[0246] Examples of embodiments of the present invention include, for example, adding a portion of the configuration of one embodiment to another embodiment, and replacing a portion of the configuration of one embodiment with a portion of the configuration of another embodiment.

[0247] The photoelectric conversion systems described in the eighth and ninth embodiments above indicate examples of photoelectric conversion systems to which photoelectric conversion devices can be applied, and photoelectric conversion systems to which photoelectric conversion devices according to embodiments of the present invention can be applied are not limited to... Figure 23 , Figure 24A and Figure 24B The configuration is illustrated in the diagram. This also applies to the TOF system described in the tenth embodiment, the endoscope described in the eleventh embodiment, and the smart glasses described in the twelfth embodiment.

[0248] Each of the above embodiments merely indicates a specific example for implementing the invention, and the scope of the invention should not be understood in a limiting manner based on these. In other words, embodiments of the invention can be implemented in various forms without departing from its technical concept or main features.

[0249] This disclosure includes the following configurations.

[0250] (Configuration 1)

[0251] A processing apparatus includes: a first storage unit for storing first array data based on output values ​​of a plurality of pixels arranged in an array; and a second storage unit having second array data stored therein, the second array data being used to correct the output values ​​from the plurality of pixels. The processing apparatus further includes a correction unit comprising a calculation unit configured to correct the output value of at least one of the plurality of pixels based on the first array data and the second array data.

[0252] (Configuration 2)

[0253] The processing apparatus according to configuration 1 further includes: a defect extraction unit configured to extract the output value and position of a defective pixel from the first array data; and an array generation unit configured to generate third array data based on the second array data and the output value and position of the defective pixel. Furthermore, a calculation unit corrects the output value of at least one peripheral pixel located around the defective pixel based on the first array data and the third array data.

[0254] (Configuration 3)

[0255] According to the processing apparatus of configuration 1, the correction unit includes a defect correction unit, and the defect correction unit corrects defective pixels in the fourth array data generated by the computing unit.

[0256] (Configuration 4)

[0257] According to the processing apparatus described in configuration 2 or 3, the second array data includes at least three data blocks, and the one-dimensional data corresponding to at least one row or column of the second array data has a peak or bottom value at the center.

[0258] (Configuration 5)

[0259] According to the processing apparatus described in configuration 4, the one-dimensional data has a distribution that monotonically changes from the peak or bottom value toward the data ends.

[0260] (Configuration 6)

[0261] According to the processing apparatus described in configuration 4 or 5, the different one-dimensional data that share the peak or bottom value of the one-dimensional data and are arranged in the direction intersecting with the one-dimensional data have a distribution that monotonically changes from the peak or bottom value of the different one-dimensional data toward the data ends.

[0262] (Configuration 7)

[0263] According to any one of configurations 2 to 6, the processing apparatus, wherein the first array data comprises N rows and M columns of data, wherein one of N and M is an integer greater than or equal to 2, and the other of N and M is an integer greater than or equal to 1.

[0264] (Configuration 8)

[0265] According to the processing apparatus described in configuration 2, the second storage unit has multiple types of second array data stored therein, and the array generation unit selects the type of second array data to be used based on the environment in which image capture is performed.

[0266] (Configuration 9)

[0267] According to the processing apparatus described in configuration 2, the third array data is generated by multiplying the second array data and the array data generated based on the output value of the defective pixel.

[0268] (Configuration 10)

[0269] According to the processing apparatus described in configuration 3, the fourth array data is generated by dividing the first array data and the second array data.

[0270] (Configuration 11)

[0271] According to any one of configurations 2 to 10, the processing apparatus wherein the defect extraction unit extracts defect pixels based on address data of the defect pixels that have been acquired in advance.

[0272] (Configuration 12)

[0273] According to any one of the configurations 2 to 11, in the processing apparatus, when the difference between the output value of the first pixel and the output value of the second pixel adjacent to the first pixel is greater than or equal to a fixed threshold, the defect extraction unit determines the first pixel or the second pixel as a defect pixel.

[0274] (Configuration 13)

[0275] According to any one of configurations 2 to 12, the processing apparatus wherein, when the ratio between the output value of the first pixel and the output value of the second pixel adjacent to the first pixel is greater than or equal to a fixed threshold, the defect extraction unit determines the first pixel or the second pixel as a defect pixel.

[0276] (Configuration 14)

[0277] According to any one of configurations 2 to 13, the processing apparatus wherein the defect extraction unit determines pixels having an output value greater than or equal to a fixed threshold as defect pixels.

[0278] (Configuration 15)

[0279] According to any one of configurations 2 to 14, the processing apparatus wherein the gain adjustment unit configured to apply digital gain to the output values ​​of the plurality of pixels is set as a pre-unit of the first storage unit.

[0280] (Configuration 16)

[0281] According to any one of configurations 2 to 15, the processing apparatus includes a nonlinear correction unit configured to perform nonlinear correction of the first array data.

[0282] (Configuration 17)

[0283] According to the processing apparatus described in configuration 16, the computing unit generates correction array data based on first array data and second array data obtained before performing nonlinear correction. The computing unit corrects the output value of at least one pixel among the plurality of pixels based on the first array data and correction array data obtained after performing nonlinear correction.

[0284] (Configuration 18)

[0285] According to the processing apparatus described in configuration 17, the second array data includes at least three data blocks, and the center value of the one-dimensional data corresponding to at least one row or column of the second array data is zero.

[0286] (Configuration 19)

[0287] According to any one of configurations 2 to 18, the processing apparatus, wherein the correction unit includes a demosaic unit as a subsequent unit to the calculation unit.

[0288] (Configuration 20)

[0289] According to any one of configurations 2 to 19, the processing apparatus performs correction of the output values ​​of the plurality of pixels in a second frame acquired later than the first frame, based on the output values ​​of defective pixels or address data of defective pixels extracted from the output values ​​of the plurality of pixels in the first frame.

[0290] (Configuration 21)

[0291] According to any one of configurations 2 to 20, the processing apparatus selects whether to perform correction in the computing unit based on the signal level of the subject or the temperature of the plurality of pixels.

[0292] (Configuration 22)

[0293] According to any one of configurations 2 to 21, the processing apparatus generates second array data based on a pattern of defective pixels in the OB region of the plurality of pixels.

[0294] (Configuration 23)

[0295] According to the processing apparatus described in configuration 2, after the output values ​​of the surrounding pixels have been corrected by the calculation unit, the output values ​​of the defective pixels are corrected based on the corrected output values ​​of the surrounding pixels.

[0296] (Configuration 24)

[0297] According to any one of configurations 2 to 23, the processing apparatus wherein the defect extraction unit extracts defect pixels by pattern matching.

[0298] (Configuration 25)

[0299] A photoelectric conversion system includes a processing device according to any one of configurations 1 to 24, and the plurality of pixels configured to output first array data.

[0300] (Configuration 26)

[0301] According to the photoelectric conversion system of configuration 25, each of the plurality of pixels includes an avalanche photodiode.

[0302] According to embodiments of the present invention, defect correction accuracy and image quality can be improved.

[0303] Although the invention has been described with reference to embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but is defined by the scope of the appended claims.

Claims

1. A processing apparatus, comprising: The first storage unit is used to store first array data based on the output values ​​of a plurality of pixels arranged in an array; The second storage unit has a second array of data stored therein, which is to be used for correction of the output values ​​from the plurality of pixels; The defect extraction unit is configured to extract the output value and position of the defective pixel from the first array data; An array generation unit is configured to generate third array data based on the second array data and the output value and position of the defective pixel; as well as The correction unit includes a calculation unit configured to correct the output value of at least one peripheral pixel located around the defective pixel based on the first array data and the third array data.

2. The processing apparatus according to claim 1, in, The correction unit includes a defect correction unit, and The defect correction unit is configured to correct defective pixels in the fourth array data generated by the computing unit.

3. The processing apparatus according to claim 1, in, The second array data includes at least three data blocks, and Among them, the one-dimensional data corresponding to at least one row or column of the second array data has a peak or bottom value at the center.

4. The processing apparatus according to claim 3, wherein, The one-dimensional data has a distribution that monotonically changes from the peak or the bottom value toward the ends of the data.

5. The processing apparatus according to claim 3, wherein, Different one-dimensional data that share the peak or bottom value of the one-dimensional data and are arranged in the direction intersecting with the one-dimensional data have a distribution that monotonically changes from the peak or bottom value of the different one-dimensional data toward the data ends.

6. The processing apparatus according to claim 1, wherein, The first array data comprises N rows and M columns of data, where one of N and M is an integer greater than or equal to 2, and the other of N and M is an integer greater than or equal to 1.

7. The processing apparatus according to claim 1, in, The second storage unit has multiple types of second array data stored therein, and The array generation unit is configured to select the type of the second array data to be used based on the environment in which the image capture is performed.

8. The processing apparatus according to claim 1, wherein, The third array data is generated by multiplying the second array data and the array data generated based on the output value of the defective pixel.

9. The processing apparatus according to claim 2, wherein, The fourth array data is generated by dividing the first array data and the second array data.

10. The processing apparatus according to claim 1, wherein, The defect extraction unit extracts defect pixels based on the address data of the defect pixels that have been acquired in advance.

11. The processing apparatus according to claim 1, wherein, If the difference between the output value of the first pixel and the output value of the second pixel adjacent to the first pixel is greater than or equal to a fixed threshold, the defect extraction unit determines the first pixel or the second pixel as a defective pixel.

12. The processing apparatus according to claim 1, wherein, If the ratio between the output value of the first pixel and the output value of the second pixel adjacent to the first pixel is greater than or equal to a fixed threshold, the defect extraction unit determines the first pixel or the second pixel as a defective pixel.

13. The processing apparatus according to claim 1, wherein, The defect extraction unit identifies pixels with output values ​​greater than or equal to a fixed threshold as defective pixels.

14. The processing apparatus according to claim 1, wherein, A gain adjustment unit configured to apply digital gain to the output values ​​of the plurality of pixels is set as a pre-unit of the first storage unit.

15. The processing apparatus according to claim 1, wherein, The correction unit includes a nonlinear correction unit configured to perform nonlinear correction of the first array data.

16. The processing apparatus according to claim 15, wherein, The computing unit is configured to generate correction array data based on first array data and second array data obtained before performing the nonlinear correction, and to correct the output value of at least one of the plurality of pixels based on the first array data and the correction array data obtained after performing the nonlinear correction.

17. The processing apparatus according to claim 16, in, The second array data includes at least three data blocks, and The center value of the one-dimensional data corresponding to at least one row or column of the second array data is zero.

18. The processing apparatus according to claim 1, wherein, The correction unit includes a demosaic unit, which is a subsequent unit to the calculation unit.

19. The processing apparatus according to claim 1, wherein, Based on the output value of the defective pixel or the address data of the defective pixel extracted from the output values ​​of the plurality of pixels in the first frame, correction is performed on the output values ​​of the plurality of pixels in the second frame acquired later than the first frame.

20. The processing apparatus according to claim 1, wherein, Whether to perform correction in the computing unit is selected based on the signal level of the subject or the temperature of the plurality of pixels.

21. The processing apparatus according to claim 1, wherein, The second array data is generated based on a pattern of defective pixels in the optical black (OB) region of the plurality of pixels.

22. The processing apparatus according to claim 1, wherein, After the output values ​​of the surrounding pixels have been corrected by the calculation unit, the output value of the defective pixel is corrected based on the corrected output values ​​of the surrounding pixels.

23. The processing apparatus according to claim 1, wherein, The defect extraction unit is configured to extract defect pixels through pattern matching.

24. A photoelectric conversion system, comprising: The processing apparatus according to any one of claims 1 to 23; as well as The plurality of pixels configured to output the data of the first array.

25. The photoelectric conversion system according to claim 24, wherein, Each of the plurality of pixels includes an avalanche photodiode.

Citation Information

Patent Citations

  • Correction of cluster defects in imagers

    US20070030365A1

  • Method and apparatus for image processing using weighted defective pixel replacement

    US20040096125A1