Imaging apparatus, processing method, and computer program

Abstract

To provide an imaging apparatus capable of performing defect correction which is hardly affected by a subject when there is a pixel defect.SOLUTION: An image pickup apparatus includes: an image pickup element including a plurality of pixels each including a photoelectric conversion unit, a first charge holding unit configured to hold an output of the photoelectric conversion unit, and a second charge holding unit configured to hold an output of the photoelectric conversion unit; and a defective pixel correction unit configured to calculate an interpolation pixel value for interpolating a pixel value from the first charge holding unit based on a pixel value output from the second charge holding unit of a predetermined pixel when the first charge holding unit of the predetermined pixel has a defect.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to an imaging device, a processing method, a computer program, and the like. [Background technology]

[0002] Among so-called CMOS sensors, there is a GS sensor that has a charge storage section in each pixel, providing a global shutter (GS) function. The pixels of this GS sensor have a gate that transfers the signal charge accumulated in the photoelectric conversion section to the charge storage section.

[0003] In a GS sensor, the GS function is realized by basically transferring charge from the photoelectric conversion unit to the charge storage unit for all pixels at the same time, and by making the start and end timing of charge accumulation in the photoelectric conversion unit the same for all pixels.

[0004] Furthermore, by configuring multiple charge storage units for one photoelectric conversion unit and transferring charges to each charge storage unit multiple times during one frame period, it is possible to obtain multiple images with different total charge accumulation times for the charges transferred to each charge storage unit.The dynamic range can then be improved by combining the multiple images obtained (Patent Document 1).

[0005] On the other hand, defects can occur in the image sensors used in CMOS sensors during the manufacturing process. For example, if there is a defect in the charge storage section, electrons will leak into the charge storage section depending on the time the charge is stored in the charge storage section, and a signal will be output that is the sum of the charge stored in the charge storage section and the amount of electron leakage. This will cause the output level to be higher than that of other normal pixels, resulting in degradation of image quality.

[0006] Therefore, if there is a defect in the charge storage section used for long-time accumulation in the GS sensor of Patent Document 1, there will be a signal difference equivalent to the accumulation time ratio with respect to the signal from the charge storage section used for short-time accumulation, and the signal from the charge storage section accumulated for long times due to the defect will be an excessive output.As a result, when the image signals from long-time accumulation and short-time accumulation are combined, the defect will remain.On the other hand, in Patent Document 2, defective pixels are corrected using a calculated value interpolated from surrounding pixels. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] US Patent Application Publication No. 2013 / 0135486 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-044452 Summary of the Invention [Problem to be solved by the invention]

[0008] However, the technology in Patent Document 2 performs interpolation processing on defective pixels using signals from surrounding pixels regardless of the location of the defect. Therefore, for subjects with high contrast or spatial frequency, interpolated pixel values ​​cannot be generated correctly from the surrounding pixels, and correction marks may remain, resulting in degradation of image quality. SUMMARY OF THE INVENTION An object of the present invention is to solve the above-mentioned problems and to provide an imaging device that, when there is a pixel defect, is capable of correcting the defect in a manner that is less susceptible to the influence of the subject. [Means for solving the problem]

[0009] An imaging device according to one aspect of the present invention comprises: a photoelectric conversion unit; a first charge holding unit for holding an output of the photoelectric conversion unit; a second charge holding unit for holding an output of the photoelectric conversion unit; an imaging element having a plurality of pixels each having and a defective pixel correction unit that, when there is a defect in the first charge storage unit of a specified pixel, calculates an interpolated pixel value for interpolating the pixel value from the first charge storage unit based on the pixel value output from the second charge storage unit of the pixel. [Effects of the Invention]

[0010] According to the present invention, an imaging device can be obtained that, when there is a pixel defect, is capable of correcting the defect without being affected by the subject. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of a schematic configuration of an image sensor 100 according to a first embodiment. [Figure 2] 1 is a functional block diagram showing an example of the configuration of an imaging device 200 according to a first embodiment. [Figure 3] 2 is an equivalent circuit diagram of each photoelectric conversion pixel of the imaging device according to the first embodiment. FIG. [Figure 4] 1A and 1B are diagrams illustrating an example of defective pixel correction according to the first embodiment. [Figure 5] 5A and 5B are diagrams illustrating an example of a change in the amount of electric charge in defective pixel correction according to the first embodiment. [Figure 6] 10A and 10B are diagrams illustrating an example of a change in the amount of electric charge in defective pixel correction according to the second embodiment. [Figure 7] 10A and 10B are diagrams illustrating another example of the transition of the amount of charge in defective pixel correction according to the second embodiment. [Figure 8] FIG. 10 is an equivalent circuit diagram of an imaging device according to a third embodiment. [Figure 9] 10 is a schematic diagram of a cross section of each pixel of an imaging device according to a third embodiment. FIG. [Figure 10] 10A and 10B are diagrams illustrating an example of defective pixel correction according to the third embodiment. [Figure 11] 10A and 10B are diagrams illustrating an example of pixel correction when a charge storage portion has a defect in the fourth embodiment. [Figure 12] FIG. 10 is an equivalent circuit diagram of an imaging device according to a fourth embodiment. [Figure 13] 10A and 10B are diagrams illustrating an example of pixel correction when an FD has a defect, according to a fifth embodiment. [Figure 14] 13 is a flowchart illustrating an example of a processing method according to the sixth embodiment. [Figure 15] 13 is a flowchart illustrating an example of correction processing when a charge holding portion has a defect in the sixth embodiment. [Figure 16] 13 is a flowchart showing an example of correction processing when there is a defect in the FD area in the sixth embodiment. [Figure 17] 13A and 13B are diagrams illustrating an example of defective pixel correction according to the seventh embodiment. [Figure 18] 13A and 13B are diagrams illustrating an example of defective pixel correction according to the eighth embodiment. [Figure 19] 13A and 13B are diagrams illustrating an example of defective pixel correction according to a ninth embodiment. [Figure 20] 20 is a flowchart illustrating an example of a processing method according to a tenth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions will be omitted or simplified.

[0013] In the following embodiments, the signal carriers are electrons, the signal storage layer is an N-type semiconductor, and the transistors forming the circuits are N-type MOS transistors unless otherwise specified. However, this is not limiting, and the signal carriers may be holes, and P-type may be used instead of the above N-type.

[0014] In the following embodiments, an example will be described in which a GS sensor is used as the imaging device. Each pixel (each photoelectric conversion pixel) in the embodiments has a photoelectric conversion unit, a charge storage unit, a photoelectric conversion unit charge transfer MOS transistor for transferring the signal charge of the photoelectric conversion unit to the charge storage unit, and an amplification MOS transistor for amplifying and outputting the signal charge. Each pixel also has a charge storage unit charge transfer MOS transistor for transferring the signal charge of the memory to the amplification MOS transistor.

[0015] <Embodiment 1> 1 is a block diagram showing a schematic configuration of an image sensor 100 according to embodiment 1. The image sensor 100 includes a pixel unit 101, a vertical scanning circuit 102, a column amplifier circuit 103, a horizontal scanning circuit 104, an output circuit 105, and a control circuit 106.

[0016] The pixel unit 101 includes a plurality of pixels (photoelectric conversion pixels) 107 arranged two-dimensionally including a plurality of rows and a plurality of columns. That is, the image sensor 100 includes a plurality of pixels (photoelectric conversion pixels) 107. The vertical scanning circuit 102 supplies control signals to a plurality of transistors included in the pixels 107, and controls the on (conductive state) or off (non-conductive state) of these transistors.

[0017] A column signal line 108 is provided for each column of pixels 107, and signals from the pixels 107 are read out to the column signal line 108 for each column. The column amplifier circuit 103 includes an amplifier for amplifying the pixel signals output to the column signal line 108, and an AD conversion circuit for analog-to-digital conversion of the signals. The horizontal scanning circuit 104 supplies a control signal to a switch connected to the amplifier of the column amplifier circuit 103, and controls the switch to be on or off.

[0018] The control circuit 106 controls the vertical scanning circuit 102, the column amplifier circuit 103, and the horizontal scanning circuit 104. The output circuit 105 includes a buffer amplifier, a differential amplifier, etc., and outputs the pixel signal from the column amplifier circuit 103 to a signal processing unit outside the image sensor.

[0019] Fig. 2 is a functional block diagram showing an example of the configuration of the imaging device 200 according to embodiment 1. Note that some of the functional blocks shown in Fig. 2 are realized by causing a CPU or the like serving as a computer included in the imaging device to execute a computer program stored in a memory serving as a storage medium.

[0020] However, some or all of these functions may be implemented by hardware. Examples of hardware that can be used include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, the functional blocks shown in Figure 2 do not have to be built into the same housing, and may be configured as separate devices connected to each other via signal paths.

[0021] The imaging device 200 shown in FIG. 2 includes the image sensor 100, a lens 202 that forms an optical image of a subject on the light receiving part of the image sensor, and a signal processing unit 208 that processes the output signal output from the image sensor.

[0022] The signal processing unit 208 generates image data from the digital signals output by the image sensor 100, performs various corrections, compression, etc. as necessary, and outputs the image data. The signal processing unit 208 also functions as a synthesis processing unit that synthesizes signals read from the image sensor and generates a high dynamic range image. However, the above synthesis processing may be performed outside the image sensor.

[0023] The imaging device 200 further includes a RAM 215, a ROM 216, an external interface unit (external I / F unit) 212 for communicating with an external computer or the like, and a recording medium 214 made up of a semiconductor memory or the like for recording and reading image data.

[0024] Reference numeral 218 denotes an overall control calculation unit that controls the entire imaging device 200 and has a CPU as a computer. RAM 215 temporarily stores calculation results and output signals from the signal processing unit, and ROM 216 stores defective pixel data and various adjustment values, etc. The defective pixel data includes the position, level, and type of defect of the corresponding pixel. ROM 216 also stores computer programs to be executed by the CPU.

[0025] The imaging device 200 also has a defective pixel correction unit 217 for correcting the pixel values ​​of defective pixels stored in the ROM 216. The defective pixel correction unit 217 will be described in detail later.

[0026] In this embodiment, defective pixel data is pre-stored in ROM 216. However, a configuration may also be adopted in which a separate defective pixel detection block is provided to detect defective pixels and generate defective pixel data, or the defective pixel data stored in ROM 216 is updated using the detection results from the defective pixel detection block.

[0027] 3 is an equivalent circuit diagram of each photoelectric conversion pixel of the image sensor according to the first embodiment. PD1 represents a photodiode as an example of a photoelectric conversion unit. GS1_L20 and GS1_S21 are charge transfer units capable of transferring signal charges generated by the photoelectric conversion unit PD1 to subsequent circuit elements. MEM1_L22 and MEM1_S23 are charge storage units capable of storing signal charges generated by the photoelectric conversion units.

[0028] Here, MEM1_L22 functions as a first charge holding unit for holding the output of the photoelectric conversion unit, and MEM1_S23 functions as a second charge holding unit for holding the output of the photoelectric conversion unit. In this embodiment, the charge accumulated in the photoelectric conversion unit for a first exposure time is held in the first charge holding unit, and the charge accumulated in the photoelectric conversion unit for a second exposure time that is shorter than the first exposure time is held in the second charge holding unit.

[0029] In addition, GS1_L20 functions as a first charge transfer unit that transfers charges from the photoelectric conversion unit to the first charge storage unit, and GS1_S21 functions as a second charge transfer unit that transfers charges from the photoelectric conversion unit to the second charge storage unit.

[0030] TX1_L28 and TX1_S29 are transfer units, each capable of transferring signal charges held in a charge holding unit in the preceding stage to a circuit element in the succeeding stage.

[0031] FD14 is, for example, a floating diffusion region (floating diffusion area or FD region) arranged on a semiconductor substrate, which can hold signal charges transferred from a charge storage unit in the preceding stage via a transfer unit, and serves as an input node for SF16 in the following stage. That is, the floating diffusion region is configured to transfer charges from the first charge transfer unit and the second charge storage unit.

[0032] RES15 is a reset unit that can supply a reference voltage to the input node FD14 of the amplifier unit. SF16 is an amplifier unit such as a source follower circuit using MOS transistors, which amplifies a signal based on the signal charge transferred to FD14 and reads it out to the outside.

[0033] In SF16, the gate of the MOS transistor is electrically connected to FD 14. In the figure, multiple transfer units, TX1_L28 and TX1_S29, share the input node FD14 and the amplifier unit SF16, but the circuit configuration may not be such that they are shared.

[0034] SEL17 is a selection unit that is selected by a selection signal from the vertical scanning circuit 102 and is capable of reading out signals to the outside for each pixel or each pixel row. OFG18 is a charge discharge control unit that can discharge the signal charge of the photoelectric conversion unit PD1. For example, a MOS transistor can be used as the charge discharge control unit.

[0035] In this embodiment, the source is a semiconductor region of the same polarity as the signal charge that constitutes part of the photoelectric conversion unit, and the drain is a semiconductor region (overflow drain region: OFD region) to which the power supply voltage VDD 19 is supplied. Also, each of the transfer unit, reset unit, selection unit, and charge discharge control unit can use a MOS transistor.

[0036] When the charge transfer unit GS1_L20 is turned on, it transfers the signal charge generated by the photoelectric conversion unit PD1 to the charge holding unit MEM1_L22. When the charge transfer unit GS1_S21 is turned on, it transfers the signal charge generated by the photoelectric conversion unit PD1 to the charge holding unit MEM1_S23.

[0037] Note that either GS1_L20 or GS1_S21 may be turned on during photoelectric conversion in PD1. That is, GS1_L20 may be turned on and GS1_S21 may be turned off during long-time (long-time) photoelectric conversion in PD1. Conversely, GS1_L20 may be turned off and GS1_S21 may be turned on during short-time (short-time) photoelectric conversion in PD1.

[0038] When the charge transfer unit TX1_L28 is turned on, it transfers the signal charge held in the charge holding unit MEM1_L22 to the FD 14. When the charge transfer unit TX1_S29 is turned on, it transfers the signal charge held in the charge holding unit MEM1_S23 to the FD 14.

[0039] In this way, by providing two charge storage units for storing the transferred signal charge for one photoelectric conversion unit PD1, charges accumulated for a long time and charges accumulated for a short time can be stored in the respective charge storage units. Therefore, by later combining the signals based on both charges according to the brightness level, it becomes possible to obtain an image with a high dynamic range.

[0040] 4A and 4B are diagrams illustrating an example of defective pixel correction according to the first embodiment, showing an image plane output from the image sensor. Fig. 4A shows the image plane of a long-time image (an image corresponding to charges accumulated for a long time) accumulated in the charge holding unit MEM1_L22 in Fig. 3.

[0041] Fig. 4(B) shows an image plane of a short-time image (an image corresponding to charges accumulated for a short time) accumulated in the charge storage unit MEM1_S23 in Fig. 3. Fig. 4 shows an example of an output image of a GS sensor in which color filters are arranged in a Bayer array so that 401 and 411 are R (red) pixels, 402 and 412 are G (green) pixels, and 403 and 413 are B (blue) pixels, but a different color filter array may also be used.

[0042] Note that the Bayer array refers to a color filter array in which, for example, in the case of three color filters of R, G, and B, color filters of R, G, R, G, ... are arranged in each pixel of a given row, and color filters of G, B, G, B, ... are arranged in the adjacent row.

[0043] In this embodiment, a defective pixel 404 is assumed to have a defect in, for example, a charge storage portion within the pixel. In this case, for example, in a long-exposure image, an interpolated pixel value is generated for the defective pixel 404 based on the output values ​​(reference pixel values) of surrounding pixels of the same color in the same image plane, and the output value of the defective pixel 404 is replaced with the interpolated pixel value, thereby correcting the defective pixel.

[0044] However, when the subject has high contrast and spatial frequency, the reference pixel values ​​may vary greatly, making it impossible to generate the correct interpolated pixel values. In such cases, correction marks may appear, resulting in degradation of image quality.

[0045] In this embodiment, instead of or in addition to calculating an interpolated pixel value from surrounding pixels, an interpolated pixel value is calculated using pixel values ​​output from the same PD via different paths, thereby performing satisfactory defective pixel correction. As a more specific example, a case where the charge holding unit MEM1_L22 of the defective pixel 404 is defective will be described later.

[0046] 4A and 4B, the pixel signals of the defective pixel 404 and the pixel 414 are both output from the same PD but via different output paths. That is, the pixel value of the defective pixel 404 is stored in the charge storage unit MEM1_L22 and output via the FD14, whereas the pixel value of the pixel 414 is stored in the charge storage unit MEM1_S23 and output via the FD14.

[0047] For this reason, the defective pixel 404 is affected by a defect in, for example, the charge storage unit MEM1_L22, but the short-time image does not use the charge storage unit MEM1_L22 and is therefore not affected by the defect, and a normal output pixel value is obtained. Therefore, in this embodiment, an interpolated pixel value for the defective pixel 404 is generated using the pixel value of pixel 414.

[0048] If there is no defect, the pixel value of the defective pixel 404 corresponds to the charge accumulated over a long accumulation time (photoelectric conversion time) in the photoelectric conversion unit PD1, and the pixel value of the pixel 414 corresponds to the charge accumulated over a short accumulation time (photoelectric conversion time) in the same photoelectric conversion unit PD1.

[0049] The same applies to the case where GS1_L20 is turned on and GS1_S21 is turned off during long-time (long-time) photoelectric conversion in PD1, and GS1_L20 is turned off and GS1_S21 is turned on during short-time (short-time) photoelectric conversion in PD1. However, in this case, the effect of defects in the charge storage section becomes even more pronounced.

[0050] That is, the pixel value of the defective pixel 404 and the pixel value of the pixel 414 are signals that differ in actuality by the ratio of the accumulation times (photoelectric conversion times) in the photoelectric conversion unit PD1. In the description of this embodiment, the accumulation time is used to mean the same as the photoelectric conversion time or the exposure time.

[0051] Therefore, the corrected pixel value of defective pixel 404 is calculated as the pixel value of pixel 414 × (long exposure time accumulation time in photoelectric conversion unit PD1 (first exposure time) / short exposure time accumulation time in photoelectric conversion unit PD1 (second exposure time)). That is, the defective pixel correction unit 217 calculates an interpolated pixel value based on the pixel value read from the second charge holding unit, in accordance with the ratio between the first exposure time and the second exposure time.

[0052] In this way, by calculating an interpolated pixel value using pixel values ​​output from the same PD via different paths, it is possible to generate a defective pixel value without being affected by surrounding pixels. This enables effective defective pixel correction, particularly for subjects with high contrast and spatial frequency.

[0053] As described above, the video signal processing unit of the imaging device of this embodiment accumulates signal charges accumulated at different exposure times (photoelectric conversion times) in multiple charge storage units, outputs a long-exposure image and a short-exposure image, and combines them to obtain an image with a high dynamic range. Therefore, in this embodiment, defective pixel correction using pixel values ​​output via different paths is performed by the defective pixel correction unit before the combination process of combining the long-exposure image and the short-exposure image.

[0054] <Embodiment 2> In the first embodiment, we demonstrated that correction is performed using pixel signals from different paths connected to a common photoelectric conversion unit, thereby generating interpolated pixel values ​​without being affected by surrounding pixels, enabling good defective pixel correction, particularly for subjects with high contrast and spatial frequency. In the second embodiment, we describe the relationship between the synthesis process that achieves a high dynamic range and the interpolation process for defective pixels.

[0055] First, we will discuss the case where a defect exists in a long-time image. Fig. 5 is a graph illustrating an example of the change in the amount of charge during defective pixel correction in embodiment 1, with the horizontal axis representing the amount of incident light (brightness) and the vertical axis representing the amount of charge. The solid line represents the amount of charge accumulated in the long-time charge storage unit, and the dashed-dotted line represents the amount of charge accumulated in the short-time charge storage unit.

[0056] Figure 5 shows that the amount of charge increases in proportion to the amount of incident light, and the slope is proportional to the accumulation time. For example, if the amount of incident light remains the same, if the accumulation time is doubled, the amount of charge accumulated in the charge storage section will double.

[0057] In Figure 5, a, b, and c delimit the ranges of incident light intensity for convenience. a is the low-luminance region where the incident light intensity is low, c is the high-luminance region where the incident light intensity is high, and b is the intermediate-luminance region.

[0058] First, looking at the long-exposure image, we can see that there is no saturation in either the low- or medium-brightness areas (a) and (b), and a valid pixel signal is obtained. In the high-brightness area (c), there is a lot of charge saturation, and the pixel signal at this time is in a so-called blown-out state.

[0059] Even in this high-brightness region c, the short-exposure image is not saturated, so by combining the images in such a way that the short-exposure image is used in the high-brightness region c and the long-exposure image is used in the low-brightness region a, an image with a high dynamic range can be obtained. This is an example of HDR combining.

[0060] Here, we have described switching between long-time and short-time images at the point where the long-time image becomes saturated, but for example, in the intermediate brightness region b, the pixel signals of the long-time image and the short-time image may be weighted and added according to the amount of incident light.

[0061] 6 is a diagram illustrating an example of the charge amount transition in defective pixel correction in embodiment 2, showing an example of the charge amount relative to the amount of incident light when there is a defect in the charge storage section for obtaining a long-time image. Compared to FIG. 5, the charge amount of the long-time image has an offset caused by, for example, a defect in the charge storage section.

[0062] As before, the image is divided into areas a, b, and c according to the amount of incident light. Area a is a low-brightness area, and because long-exposure images are used preferentially in HDR compositing, the output signal is affected by defects.

[0063] In other words, this is an area where defective pixel correction is required. In the intermediate brightness area of ​​b, the original pixel signal is saturated due to an offset caused by the defect. At this time, the long-exposure image is saturated, so the short-exposure image is used preferentially.

[0064] Therefore, although there are defects, defective pixel correction is unnecessary by using the short-exposure image.In this way, when there are defects only in the long-exposure image, defective pixel correction that generates corrected pixel values ​​using the short-exposure image is effective in low-brightness areas.

[0065] Next, a case where only the short-time image has a defect will be described. Fig. 7 is a diagram illustrating another example of the change in the amount of charge during defective pixel correction in embodiment 2. Compared to Fig. 5, an offset occurs in the amount of charge in the defective pixel of the short-time image.

[0066] In the low-brightness area a and the medium-brightness area b in Figure 7, the long-exposure image is not saturated, so the long-exposure image is used for HDR compositing. Therefore, even if there are defects in the short-exposure image, it does not affect the composite image. In the high-brightness area c in Figure 7, the long-exposure image is saturated, so the short-exposure image is used.

[0067] In high-brightness areas, defects in short-time images have an effect, making defective pixel correction necessary. However, in high-brightness areas, even if defective pixel correction is performed using pixel signals output via different paths from a common photoelectric conversion unit as in embodiment 1, it is not possible to generate correct corrected pixel values ​​because long-time images are saturated.

[0068] Therefore, to solve the above problems, a method for driving the image sensor in embodiment 2 will be described. In the equivalent circuit diagram of Fig. 3, the charge retention unit MEM1_L22 is used as the charge retention unit for long seconds, and the charge retention unit MEM1_S23 is used as the charge retention unit for short seconds.

[0069] In the second embodiment, when MEM1_S23, which is the charge holding unit for short seconds, is defective, the roles of MEM1_L22 and MEM1_S23 are interchanged, and MEM1_S23 is used as the charge holding unit for long seconds, and MEM1_L22 is used as the charge holding unit for short seconds.

[0070] That is, when a defect is detected in the second charge holding unit, the charge accumulated in the photoelectric conversion unit for the first exposure time is held in the second charge holding unit, and the charge accumulated in the photoelectric conversion unit for the second exposure time, which is shorter than the first exposure time, is held in the first charge holding unit.

[0071] This causes a defect in the charge storage section for long seconds, and furthermore, since MEM1_L22 is not saturated, defective pixel correction is possible using the output pixel value of MEM1_L22 as described in the first embodiment.

[0072] In this case, the roles of the charge storage units are swapped for only some of the defective pixels. Alternatively, the pixel signals of the long-exposure image and the short-exposure image can be swapped by changing the readout order, and information on the roles of the charge storage units is used to perform HDR compositing.

[0073] Although the example given here is of replacing only pixels with defective charge retention units, the roles may be swapped all at once by checking the number of defective charge retention units on a row-by-row basis or for all pixels, so that the defective charge retention units are used for long-time periods.

[0074] That is, if the number of pixels in the same row whose charge storage unit for short exposures has a defective pixel is equal to or greater than a predetermined value, the charge storage unit for short exposures in that row may be switched to be used as the charge storage unit for long exposures. Alternatively, if the number of pixels in all pixels whose charge storage unit for short exposures has a defective pixel is equal to or greater than a predetermined value, the charge storage units for short exposures in all pixels may be switched to be used as the charge storage units for long exposures.

[0075] In this way, when the number of defects in the second charge storage units of the plurality of pixels is greater than the number of defects in the first charge storage units, the charges accumulated in the photoelectric conversion units for the first exposure time may be stored in the second charge storage units of the plurality of pixels. In this case, the charges accumulated in the photoelectric conversion units for the second exposure time, which is shorter than the first exposure time, may be stored in the first charge storage units.

[0076] In the first and second embodiments, defective pixel correction may be performed using pixel signals output from a common photoelectric conversion unit via different paths, and the circuit configuration may not be the same as the example shown in Fig. 3, and for example, the number of charge holding units is not limited to two. Furthermore, circuit configurations including other circuit elements are also included in these embodiments as long as defective pixel correction is performed using pixel signals output from a common photoelectric conversion unit via different paths.

[0077] <Embodiment 3> In the first and second embodiments, a method for correcting defective pixels using pixel signals output from a common photoelectric conversion unit via different paths has been described. In the third embodiment, defective pixels are corrected using pixel signals output from other adjacent photoelectric conversion units within the same pixel.

[0078] As in the first and second embodiments, the defective pixel correction means in the third embodiment does not generate an interpolated pixel value using pixel signals from surrounding pixels, and therefore enables good defective pixel correction for subjects with high contrast and spatial frequency.

[0079] Fig. 8 is an equivalent circuit diagram of an image sensor according to embodiment 3. The equivalent circuit in Fig. 8 includes a photoelectric conversion unit PD'601, charge holding units MEM1_L'622 and MEM1_S'623, as compared to the equivalent circuit shown in Fig. 3. The equivalent circuit in Fig. 8 further includes charge transfer units GS1_L'620 and GS1_S'621, transfer units TX1_L'628 and TX1_S'629, and a charge discharge control unit OFG'618.

[0080] The functions and control methods of each unit are the same as those in Fig. 3, and therefore will not be described here. Fig. 9 is a schematic cross-sectional view of each pixel (each photoelectric conversion pixel) of the image sensor according to embodiment 3. The photoelectric conversion units PD1 (first photoelectric conversion unit) and PD'601 (second photoelectric conversion unit) are arranged below the microlens ML701 and share the microlens ML701 for focusing light.

[0081] For this reason, the amount of signal charge generated by PD1 and PD'601 changes depending on the angle of light incident on the pixel. In other words, PD1 and PD'601 receive light from different exit pupils of the imaging lens. Therefore, the distance to the subject can be calculated by detecting the phase difference between the image signals obtained from multiple PD1s and the image signals obtained from multiple PD'601s.

[0082] Here, the charge holding unit MEM1_L22 functions as a first charge holding unit for holding the output of the first photoelectric conversion unit, and the charge holding unit MEM1_L'622 functions as a first charge holding unit for holding the output of the second photoelectric conversion unit.

[0083] In addition, the charge holding unit MEM1_S23 functions as a second charge holding unit for holding the output of the first photoelectric conversion unit, and the charge holding unit MEM1_S'623 functions as a second charge holding unit for holding the output of the second photoelectric conversion unit.

[0084] 10(A) and (B) are diagrams illustrating an example of defective pixel correction according to the third embodiment, showing an image plane at a long exposure time output from the image sensor having the configuration shown in Fig. 8. Fig. 10(A) shows an image plane that has undergone photoelectric conversion by the photoelectric conversion unit PD1 and is output via the long exposure time charge holding unit MEM1_L22.

[0085] 10B shows a long-exposure image plane accumulated by PD'601, which shares microlens ML701 with PD1, and output via charge storage unit MEM1_L'622. This embodiment describes an example of defective pixel correction when the charge storage unit MEM1_L22 of pixel 800 is defective. It should be noted that each pixel shown in FIGS. 10A and 10B is assumed to have a Bayer array color filter arranged therein, similar to those shown in FIGS. 4A and 4B.

[0086] Pixel 800 and pixel 810 respectively represent pixel signals in the image plane at long exposure times (long accumulation times) obtained by photoelectrically converting light passing through a common microlens by PD1 and PD'601, respectively.

[0087] When there are no defects in the charge storage units, the subject is uniform, and the parallax between PD1 (first photoelectric conversion unit) and PD'601 (second photoelectric conversion unit) is small, the output pixel value of pixel 800 and the output pixel value of pixel 810 will be close to each other. Therefore, in this embodiment, if there is a defect in the charge storage unit MEM1_L22 of pixel 800 for long exposures, for example, an interpolated pixel value for the charge storage unit MEM1_L22 of pixel 800 for long exposures is calculated based on the output pixel value of pixel 810.

[0088] That is, when the first charge storage unit (MEM1_L22) for the first photoelectric conversion unit of a given pixel is defective, an interpolated pixel value is calculated based on the pixel value output from the first charge storage unit (MEM1_L'622) for the second photoelectric conversion unit of the pixel, and the pixel value from the first charge storage unit for storing the output of the first photoelectric conversion unit is interpolated using the interpolated pixel value.

[0089] An example of calculating an interpolated pixel value in embodiment 3 will be described below. First, in the image plane of Fig. 10(A), the variance of pixel values ​​of surrounding pixels 801 to 808 of pixel 800 is calculated. Similarly, in the image plane of Fig. 10(B), the variance of pixel values ​​of surrounding pixels 811 to 818 is calculated.

[0090] Next, the difference between the two variances calculated as described above is calculated, and it is determined whether the difference is equal to or less than a predetermined threshold. If the difference between the variances is equal to or less than the threshold, the difference between the pixel values ​​output from two adjacent photoelectric conversion units that share a microlens is calculated.

[0091] Specifically, the difference in pixel value between surrounding pixel 801 and surrounding pixel 811 is calculated, followed by the difference in pixel value between pixel 802 and pixel 812. Subsequently, the pixel value differences of the other surrounding pixels are calculated in the same manner. The average value of the multiple differences thus obtained is calculated, and the value obtained by multiplying this average value by the pixel value of pixel 800 is set as the interpolated pixel value. Then, defective pixel correction processing is performed by replacing the pixel value of the defective pixel with the interpolated pixel value.

[0092] As described above, in this embodiment, defective pixel correction is performed using output pixel values ​​from other photoelectric conversion units that share a microlens with the photoelectric conversion unit corresponding to pixel 800. Note that the intention of comparing the difference in variance with a threshold is to avoid a decrease in the calculation accuracy of the interpolated pixel value, which would occur in the case of an object with a high spatial frequency or a high contrast, since the variance calculated in each image plane becomes large.

[0093] Furthermore, the purpose of calculating the average value of the differences between the output pixel values ​​of two adjacent photoelectric conversion units that share a microlens is to reduce the influence of parallax. Note that, although the present embodiment uses the variance and the average in the process of calculating the corrected pixel value, it is also possible to use the standard deviation, the median, or other statistical values.

[0094] <Embodiment 4> In the fourth embodiment, a method that enables good defective pixel correction even when surrounding pixels are used will be described. That is, in the fourth embodiment, when a defect exists in the first charge storage unit of a specific pixel, an interpolated pixel value is calculated for interpolating the pixel value from the first charge storage unit based on the pixel value output from the second charge storage unit of the pixel and the pixel values ​​of the pixels surrounding the specific pixel.

[0095] First, a correction method according to the fourth embodiment when there is a defect in the charge storage portion in the pixel structure described above will be described with reference to FIG.

[0096] 11(A) and (B) are diagrams illustrating an example of pixel correction when there is a defect in the charge storage portion in embodiment 4, where Fig. 11(A) shows an example of a long-time image and Fig. 11(B) shows an example of a short-time image. Note that each pixel shown in Fig. 11(A) and (B) is assumed to have a Bayer array color filter arranged therein, similar to Fig. 4(A) and (B).

[0097] 11(A) shows a state in which a defect has occurred in the charge storage section of the R pixel 1610 of the long exposure image. Normally, an image is generated by combining the signal of the R pixel 1610 of the long exposure image and the R pixel 1600 of the short exposure image, but in this case, because a defect has occurred in the charge storage section of the R pixel 1610 of the long exposure image, combining the signals in this state results in degradation of image quality.

[0098] Therefore, in this embodiment, the R pixel 1610 of the long-time image where a defect occurs is interpolated using information from the surrounding R pixels. To do this, first, the relationship with the surrounding pixels is calculated using the signal from the charge storage unit of the short-time image where no defect occurs, which is connected to the same photoelectric conversion unit as the charge storage unit of the long-time image where a defect occurs.

[0099] Specifically, the signal level ratios of R pixels 1601, 1602, 1603, and 1604 of the adjacent short-time images, which have the same color as the R pixel 1600 of the short-time image without defects, are calculated.

[0100] Here, the signal level of an R pixel 1600 in a short-time image without defects is short_R. The signal levels of R pixels 1601, 1602, 1603, and 1604 in the short-time image of the same color pixels on the top, bottom, left, and right are short_top_R, short_bottom_R, short_left_R, and short_right_R, respectively. The relationship between the R pixel 1600 and the surrounding R pixels 1601, 1602, 1603, and 1604 can be calculated as follows.

[0101] Ratio to the top R pixel 1601: top_cor_R=short_R / short_top_R

[0102] Ratio to the bottom R pixel 1602: bottom_cor_R=short_R / short_bottom_R

[0103] Ratio to pixel 1603 on the left: left_cor_R=short_R / short_left_R

[0104] Ratio to right pixel 1604: right_cor_R=short_R / short_right_R

[0105] Next, based on the relationship calculated here with the surrounding pixels of the short-time image, the interpolated pixel value of the R pixel 1610 in the long-time image where a defect occurs is calculated by weighted averaging as follows:

[0106] The signal level of the R pixel 1610 in the long exposure image where a defect has occurred is designated as long_R. The signal levels of the R pixels 1611, 1612, 1613, and 1614 of the long exposure image, which are the same color pixels above, below, left, and right, are designated as long_top_R, long_bottom_R, long_left_R, and long_right_R, respectively.

[0107] long_R={(top_cor_R×long_top_R) +(bottom_cor_R×long_bottom_R) +(left_cor_R × long_left_R) +(right_cor_R×long_right_R)} / 4

[0108] In this embodiment, a charge storage unit for long-time accumulation and a charge storage unit for short-time accumulation are connected to the same photoelectric conversion unit, so even if a defect occurs in either one, it is possible to perform good defective pixel correction by calculating an interpolated pixel value based on the relationship with surrounding pixels that are not defective, as described above.

[0109] In this manner, in this embodiment, when defective pixel correction is performed using pixel signals output from a common photoelectric conversion unit via different paths, an interpolated pixel value is calculated based on the pixel values ​​of the charge storage units and their surrounding pixels in paths where no defects occur, thereby enabling better defective pixel correction.

[0110] Although the case where there is a defect in the charge storage section for long-time accumulation has been described here, similar calculations can be made using pixel information of the long-time image when there is a defect in the charge storage section for short-time accumulation.

[0111] <Embodiment 5> Next, in a fifth embodiment, an example of correction processing when there is a defect in the FD area will be described.

[0112] Fig. 12 is an equivalent circuit diagram of an image sensor according to embodiment 5, showing an example configuration in which one FD region is shared by two charge storage units. In Fig. 12, a charge storage unit MEM1_S23 for short-second accumulation of the photoelectric conversion unit PD1 is connected to a charge storage unit MEM1_L1522 for long-second accumulation of the photoelectric conversion unit PD1501. Here, the photoelectric conversion unit PD1501 is the photoelectric conversion unit of the pixel adjacent to and below PD1.

[0113] In the configuration shown in Figure 12, if a defect occurs in the FD region, it can be considered the same as if defects occurred in the two charge storage units of the pixels arranged above and below, and interpolation processing can be performed. Here, we will explain the correction processing in such a case.

[0114] Figures 13(A) and (B) are figures explaining an example of pixel correction when there is a defect in the FD in embodiment 5, where Figure 13(A) shows an image plane output from the image sensor for a long-exposure image, and Figure 13(B) shows an image plane output from the image sensor for a short-exposure image.

[0115] In Fig. 13(A), an R pixel 1610 of a long-time image is shown as a pixel read out from the FD area where a defect occurs, and in Fig. 13(B), a G pixel 1620 of a short-time image is shown as a pixel read out from the same FD area where a defect occurs. Note that, while Fig. 12 shows PD1 being arranged above PD1501, Fig. 13 describes an example in which PD1 is arranged below PD15.

[0116] Since the G pixel 1620 of the short-time image and the R pixel 1610 of the long-time image are read from the same FD area where the defect occurs, combining them in this state will degrade the image. Therefore, correction processing is performed using surrounding pixel information for the G pixel 1620 of the short-time image and the R pixel 1610 of the long-time image that are connected to the FD area where the defect occurs.

[0117] The correction processing method can be the same as that used when there is a defect in the charge storage section described above, and it is possible to correct the G pixels of short-time images and the R pixels of long-time images.

[0118] First, a method for correcting the G pixel 1620 of the short-time image will be described. Specifically, the ratio of the signal levels of G pixels 1631, 1632, 1633, and 1634 of the long-time image, which are the same color as the G pixel 1630, is calculated.

[0119] The signal level of the G pixel 1630 is long_G, and the signal levels of the G pixels 1631, 1632, 1633, and 1634 on the top, bottom, left, and right are long_top_G, long_bottom_G, long_left_G, and long_right_G, respectively. The relationship between the G pixel 1630 and the surrounding G pixels 1631, 1632, 1633, and 1634 can be calculated as follows:

[0120] Ratio to the top G pixel 1631: top_cor_G=long_G / long_top_G

[0121] Ratio to the bottom G pixel 1632: bottom_cor_G=long_G / long_bottom_G

[0122] Ratio to pixel 1633 on the left: left_cor_G=long_G / long_left_G

[0123] Ratio to pixel 1634 on the right: right_cor_G=long_G / long_right_G

[0124] Next, using the relationship calculated here with the surrounding pixels of the long-time image, the interpolated pixel value of the G pixel 1620 connected to the FD area where the defect occurs is calculated by averaging as follows:

[0125] The signal level of the defective G pixel 1620 accumulated over a short period of time is defined as short_G. The signal levels of the G pixels 1621, 1622, 1623, and 1624 accumulated over a short period of time, which are pixels of the same color above, below, left, and right, are defined as short_top_G, short_bottom_G, short_left_G, and short_right_G, respectively.

[0126] short_G={(top_cor_G×short_top_G) +(bottom_cor_G×short_bottom_G) +(left_cor_G×short_left_G) +(right_cor_G×short_right_G)} / 4

[0127] Furthermore, for the other R pixel 1610 connected to the FD region where the defect occurs, the interpolated pixel value can be calculated in the same manner as described above.

[0128] In this way, embodiment 5 makes it possible to perform good correction processing using surrounding pixels by calculating interpolated pixel values ​​for each G pixel of a short-time image and each R pixel of a long-time image connected to the FD area where a defect occurs, using surrounding pixels.

[0129] <Embodiment 6> Next, the defective pixel correction process executed by the defective pixel correction unit 217 of the sixth embodiment will be described with reference to the flowcharts of Figures 14 to 16. Note that the operation of each step in the flowcharts of Figures 14 to 16 is performed sequentially by a CPU or the like serving as a computer within the imaging device 200 executing a computer program stored in memory.

[0130] FIG. 14 is a flowchart showing an example of a processing method according to the sixth embodiment, and is a flowchart for explaining an example of performing a defect type determination process for performing a correction process on a defect.

[0131] First, N pieces of defective pixel data stored in advance in ROM are read in order, starting from data number 1. To do this, in step S1701, the data number to be read is set to N=1, and in step S1702, it is determined whether the Nth defective pixel data exists.

[0132] If it is determined in step S1702 that there is defective pixel data, it is determined in step S1703 whether the type (location) of the defect is a charge retention portion. If it is determined in step S1703 that there is a defect in the charge retention portion, defect correction processing for the charge retention portion is performed in step S1704, as will be described later with reference to FIG.

[0133] If it is determined in step S1703 that the defect is not in the charge retention portion, it is determined in step S1706 whether the defect is in the FD region. If it is determined in step S1706 that the defect is in the FD region, defect correction processing for the FD region is performed as will be described later with reference to FIG. 16 in step S1707.

[0134] If it is determined in step S1706 that the defect is not in the FD region, correction processing is performed in step S1708 by interpolating the average value using the pixel values ​​of the surrounding pixels to correct the defect in the PD.

[0135] Here, step S1708 functions as a calculation step (calculation unit) that calculates an interpolated pixel value for interpolating the pixel value from the first charge storage unit based on the pixel values ​​of pixels surrounding the pixel when the first charge storage unit of a given pixel has a defect. The operation of the calculation unit is controlled according to the type of defect determined in steps S1702, S1703, S1706, etc.

[0136] After the correction process according to the defect (steps S1704, S1707, S1708) is performed, in step S1705, N=N+1 is set to read the next defective pixel data, and the process returns to step S1702, and the above-described process is repeated.

[0137] When reading of all defective pixel data is completed in step S1702 and it is determined that there is no N-th defective pixel data, the flow for defect type determination processing in FIG. 14 ends.

[0138] 14 shows the defective pixel correction processing steps performed by the defective pixel correction unit 217 in this embodiment. Furthermore, in the defective pixel correction steps, when a defect exists in the first charge storage unit of a given pixel, an interpolated pixel value is calculated based on the pixel value output from the second charge storage unit of the pixel to interpolate the pixel value from the first charge storage unit.

[0139] 15 is a flowchart illustrating an example of correction processing when there is a defect in the charge storage portion in embodiment 6, and illustrates an example of processing in step S1704. Note that the correction processing illustrated in FIG. 15 corresponds to the processing described in embodiments 1 to 4.

[0140] 15 starts, it is determined in step S1751 whether or not there is a defect in the charge holding section for short-second accumulation. If it is determined in step S1751 that there is a defect in the charge holding section for short-second accumulation, correction processing for the charge holding section for short-second accumulation is performed in step S1752.

[0141] That is, in step S1752, the signal level ratio between the long-time image and the surrounding pixels is calculated, i.e., the level ratio between the pixel signal of the long-time image obtained from the long-time accumulation charge storage unit connected to the same photoelectric conversion unit as the photoelectric conversion unit connected to the short-time accumulation charge storage unit with the defect and the pixel signals of the long-time images of the surrounding pixels is calculated.

[0142] Next, in step S1753, the interpolated pixel value of the short-time image is calculated. That is, the interpolated pixel value of the short-time image containing the defect is calculated using the signal level ratio between the pixel signal levels of the surrounding pixels in the short-time image and the pixel signal level of the long-time image calculated previously.

[0143] On the other hand, if it is determined in step S1751 that the charge holding section for short-second accumulation is not defective, it means that a defect has occurred in the charge holding section for long-second accumulation, and the process proceeds to step S1754.

[0144] In step S1754, the signal level ratio between the surrounding pixels of the short-time image is calculated for the long-time accumulation charge storage unit interpolation process. That is, in step S1754, the signal level ratio between the pixel signal of the short-time image obtained from the short-time accumulation charge storage unit connected to the same photoelectric conversion unit as the photoelectric conversion unit connected to the defective long-time accumulation charge storage unit and the pixel signals of the short-time image of the surrounding pixels is calculated.

[0145] Next, in step S1755, the interpolated pixel value of the long exposure image is calculated. That is, the interpolated pixel value of the long exposure image containing the defect is calculated using the ratio between the pixel signal levels of the surrounding pixels in the long exposure image and the pixel signal levels of the short exposure image calculated previously.

[0146] 16 is a flowchart showing an example of correction processing when there is a defect in the FD area in embodiment 6, and shows an example of processing in step S1707. Note that the correction processing shown in Fig. 16 corresponds to the processing described in embodiment 5, and when there is a defect in the FD area, processing is performed to calculate interpolated pixel values ​​of the short-time image and the long-time image connected to the FD area where the defect exists.

[0147] First, in step S1801, the signal level ratio between the long-second charge storage unit and the surrounding pixels is calculated for the charge storage unit correction process for the short-second image. That is, the signal level ratio between the pixel signal of the long-second image obtained from the charge storage unit for long-second accumulation connected to the same photoelectric conversion unit as the photoelectric conversion unit connected to the defective short-second accumulation charge storage unit and the pixel signals of the long-second image of the surrounding pixels is calculated.

[0148] Next, in step S1802, the interpolated pixel value of the short-time charge storage unit is calculated. That is, the interpolated pixel value of the short-time image where the defect occurs is calculated using the level ratio between the pixel signal levels of the surrounding pixels in the short-time image and the pixel signal level of the long-time image calculated previously.

[0149] Next, in step S1803, the signal level ratio of the short-time charge storage unit to the surrounding pixels is calculated as the charge storage unit correction process for the long-time image. That is, the level ratio of the pixel signal of the short-time image obtained from the short-time accumulation charge storage unit connected to the same photoelectric conversion unit as the defective long-time accumulation charge storage unit to the pixel signal of the short-time image of the surrounding pixels is calculated.

[0150] Next, in step S1804, the interpolated pixel value of the long-time charge storage unit is calculated. That is, the interpolated pixel value of the long-time image where the defect occurs is calculated using the level ratio between the pixel signal levels of the surrounding pixels in the long-time image and the pixel signal level of the short-time image calculated previously.

[0151] <Embodiment 7> 17A and 17B are diagrams for explaining an example of defective pixel correction according to the seventh embodiment, and are diagrams for explaining a processing example in which some of the charge storage units of the surrounding pixels used in the correction process are saturated. Note that in FIGS. 17A and 17B, the same numbers as in FIGS. 11A and 11B indicate the same components.

[0152] As described above, GS1_L20 may be turned on and GS1_S21 may be turned off during long-exposure (long-time) photoelectric conversion in PD 1, or GS1_L20 may be turned off and GS1_S21 may be turned on during short-exposure (short-time) photoelectric conversion in PD 1. In this case, when a bright subject is photographed, it is conceivable that the charge storage unit used for long-exposure accumulation may exceed the amount of charge that can be stored in the charge storage unit and become saturated.

[0153] When the charge storage section is saturated (when the pixel value is equal to or greater than a predetermined value), the correct signal level cannot be read out, and performing correction processing in this state leads to deterioration of image quality. Therefore, in such cases, correction processing is performed using pixel information from the charge storage section that is not saturated. In other words, in this embodiment, pixel values ​​of surrounding pixels that are equal to or greater than a predetermined value are not used to calculate the interpolated pixel value.

[0154] Figure 17(A) shows a case where a portion of a long-time image is saturated, with a defect occurring in the R pixel 1610 of the long-time image and the upper R pixel 1611 being saturated. Figure 17(B) shows an example of a short-time image, similar to Figure 11(B).

[0155] In this case, the relationship with surrounding pixels is calculated using the short-time image using the method described in Figure 11, and since the pixels located above the defective pixel in the long-time image are saturated, correction processing is performed excluding the saturated pixels. In other words, the interpolated pixel value of the long-time image is calculated by averaging the values ​​of the surrounding pixels excluding the saturated pixels as follows:

[0156] long_R={(bottom_cor_R×long_bottom_R) +(left_cor_R × long_left_R) +(right_cor_R×long_right_R)} / 3

[0157] <Embodiment 8> 18(A) and (B) are diagrams for explaining an example of defective pixel correction in embodiment 8, with Fig. 18(A) showing an example of a long-time exposure image and Fig. 18(B) showing an example of a short-time exposure image. In embodiment 8, a correction method will be described for the case where a defect occurs in G pixel 1620 of the short-time exposure image shown in Fig. 18(B) and G pixel 1631 of the long-time exposure image is saturated.

[0158] In this case, because it is not possible to calculate the relationship with the upper G pixel 1631 using the long-exposure image, interpolation processing is performed excluding the upper G pixel 1621. In other words, the interpolated pixel value of the short-exposure image is calculated by averaging the pixel values ​​of the surrounding G pixels excluding the upper G pixel 1621, as shown below.

[0159] short_G={(bottom_cor_G×short_bottom_G) +(left_cor_G×short_left_G) +(right_cor_G×short_right_G)} / 3

[0160] When saturated pixels exist, the interpolated pixel value can be calculated by performing correction processing excluding the saturated pixels.

[0161] <Embodiment 9> As a ninth embodiment, a correction process using information on adjacent short-time images of different colors when all pixels of the long-time images used for correction are saturated will be described with reference to FIG.

[0162] Figures 19(A) and (B) are diagrams for explaining an example of defective pixel correction in embodiment 9, with Figure 19(A) showing an example of a long-exposure image and Figure 19(B) showing an example of a short-exposure image. Figure 19(B) shows a state in which G pixel 1620 in the short-exposure image has a defect. In the correction process in this case, the method described above calculates the relationship between G pixel 1630 and its surrounding G pixels using G pixels 1630, 1631, 1632, 1633, and 1634 of the long-exposure image shown in Figure 19(A).

[0163] However, in the ninth embodiment, it is assumed that all of the G pixels 1630, 1631, 1632, 1634, and 1635 are saturated. In this state, if the correction process described above is performed, it will not be possible to perform correction with high reliability.

[0164] Therefore, in the ninth embodiment, correction processing is performed using information from a short-time image of an R pixel, which is a different color pixel adjacent to the upper side of a G pixel. Specifically, correction processing is performed using R pixel 1650 and its surrounding R pixels 1651, 1652, 1653, and 1654. In this case, the relationship between R pixel 1650 and its surrounding pixels is calculated as follows:

[0165] The signal level of the R pixel 1650 accumulated for a short time is short_R. The signal levels of the R pixels 1651, 1652, 1653, and 1654 accumulated for a short time of the same color pixels on the top, bottom, left, and right are short_top_R, short_bottom_R, short_left_R, and short_right_R, respectively.

[0166] Ratio to the top R pixel 1651: top_cor_R=short_R / short_top_R

[0167] Ratio to bottom R pixel 1652: bottom_cor_R=short_R / short_bottom_R

[0168] Ratio to the left R pixel 1653: left_cor_R=short_R / short_left_R

[0169] Ratio to the right R pixel 1654: right_cor_R=short_R / short_right_R The interpolated pixel value of the G pixel 1620 is then calculated by averaging as follows: As mentioned above, the signal level of the G pixel 1620 accumulated over a short period of time is defined as short_G. The signal levels of the G pixels 1621, 1622, 1623, and 1624 accumulated over a short period of time on the top, bottom, left, and right sides are defined as short_top_G, short_bottom_G, short_left_G, and short_right_G, respectively.

[0170] short_G={(top_cor_R×short_top_G) (bottom_cor_R×short_bottom_G) (left_cor_R×short_left_G) (right_cor_R×short_right_G)} / 4

[0171] In this way, when the pixel of the color used for correction is saturated, the correction process may be performed using information on a pixel of a different color adjacent to the defective pixel.

[0172] <Embodiment 10> So far, we have described methods for correcting defective pixels when there is a defect in the charge storage region or floating diffusion region (FD) using embodiments 1 to 9. These methods for correcting defective pixels before HDR blending processing can be broadly classified into two types.

[0173] One method uses output pixel values ​​output via different paths within the same pixel, and includes, for example, embodiments 1 to 3, and is referred to as, for example, the first defective pixel correction method. The other method calculates an interpolated pixel value using surrounding pixels, and includes, for example, embodiments 4, 5, 7 to 9, and is referred to as, for example, the second defective pixel correction method.

[0174] On the other hand, as a method for correcting defective pixels after HDR synthesis, for example, there is a common defective pixel correction method, such as that performed in step S1708, in which the median value of the pixel values ​​of surrounding pixels is used as the interpolated pixel value, which is called, for example, the third defective pixel correction method.

[0175] The third defective pixel correction method calculates a corrected pixel value from surrounding pixels, so for subjects with high contrast and spatial frequency, better defective pixel correction can be achieved by using the first or second defective pixel correction method. In the tenth embodiment, the defective pixel correction method is switched.

[0176] That is, the first defective pixel correction method corrects defects occurring in the long-second charge storage section, and the second defective pixel correction method corrects defects occurring in the charge storage section or the floating diffusion section. Therefore, the first defective pixel correction method and the second defective pixel correction method are switched depending on the location of the defect.

[0177] In addition, the first and second defective pixel correction methods perform defective pixel correction before HDR blending. Depending on the location of the defect and the pixel value, the defect may not become apparent after HDR blending, so the defective pixel correction method is switched taking these factors into consideration.

[0178] Fig. 20 is a flowchart showing an example of a processing method according to the tenth embodiment, illustrating a processing example for selecting a correction method. Note that the operation of each step in the flowchart of Fig. 20 is performed sequentially by a CPU or the like serving as a computer in the imaging device executing a computer program stored in a memory.

[0179] First, in step S901, it is determined whether the location of the defective pixel is in the long-exposure charge storage section. If there is a defect in the long-exposure charge storage section, in step S902 it is determined whether the long-exposure image is below a predetermined upper threshold. That is, it is determined whether the desired pixel of interest in the long-exposure image is below the upper threshold. This upper threshold is set as the upper limit of the brightness range used for the long-exposure image in, for example, HDR compositing processing.

[0180] In typical HDR merging processes, high-brightness areas are processed using short-exposure images, so even if there is a defect in the long-exposure image, the defect does not affect the image after HDR merging. Therefore, if it is determined in step S902 that the pixel is greater than or equal to the upper threshold, no defective pixel correction process is performed, and the flow in Figure 20 ends. On the other hand, if it is determined in step S902 that the pixel of interest in the long-exposure image is less than the upper threshold, defective pixel correction process is performed using the first defective pixel correction method in step S903, and then the flow in Figure 20 ends.

[0181] On the other hand, if it is determined in step S901 that the location of the defective pixel is not in the long-time charge storage section, it is determined in step S904 whether the location of the defective pixel is in the short-time charge storage section. If it is determined in step S904 that there is a defect in the short-time charge storage section, it is determined in step S905 whether the pixel of interest in the short-time image is equal to or greater than a predetermined lower threshold.

[0182] This lower threshold is the lower limit of the brightness range in which short-exposure images are used, for example, in HDR compositing processing. In general HDR compositing processing, long-exposure images are used for low-brightness areas, so even if there is a defect in the short-exposure image, the defect does not affect the image after HDR compositing processing. Therefore, if it is determined in step S905 that the pixel is below the lower threshold, the flow in Figure 20 ends without performing defective pixel correction processing. On the other hand, if it is determined in step S905 that the pixel of interest in the short-exposure image is equal to or greater than the lower threshold, the flow in Figure 20 ends without performing defective pixel correction processing using the second defective pixel correction method in step S906.

[0183] If it is determined in step S904 that the defective pixel is not located in the charge storage section for short seconds, it is then determined in step S907 whether the defective pixel is located in the floating diffusion section (FD). If the determination in step S907 is Yes, the defective pixel correction process is performed using the second defective pixel correction method in step S908, and the flow in FIG. 20 ends.

[0184] If it is determined in step S907 that the defective pixel is not located in the floating diffusion region, the defective pixel is corrected by the third defective pixel correction method in step S909, and the flow of FIG. 20 ends.

[0185] Next, as an example of a method for identifying the location of defective pixels, we will explain a method for acquiring an image for detecting defective pixels by turning each transfer unit on and off. Defective pixels mainly occur when electrons leak into the photoelectric conversion unit, charge storage unit, or floating diffusion unit, resulting in excessive output and appearing as so-called white defects, so it is preferable to use an image in a light-shielded state for defective pixel detection.

[0186] For example, in the configuration of Figure 3, if GS1_L20 and GS1_S21 are turned OFF, TX1_L28 is turned ON, and TX1_S29 is turned OFF, and there is no defect in the image, then if there is a defect in the image obtained by turning GS1_L20 and GS1_S21 OFF, TX1_L28 OFF, and TX1_S29 ON, it can be determined that the defect is in MEM1_L22.

[0187] Here we have shown a method for estimating defects in the long-second charge storage section, but defects in the short-second charge storage section or floating diffusion section can also be estimated from the opening and closing of the transfer section and the output results.

[0188] In the tenth embodiment, an example of a method for switching between the first to third defective pixel correction methods described above depending on the location of the defective pixel, etc. However, the imaging device is not limited to the first to third defective pixel correction methods described above, and may have four or more defective pixel correction methods, and may switch between at least two of the defective pixel correction methods.

[0189] Furthermore, the two or more (plural) defective pixel correction methods may be switched depending on at least one of the type of defect, the subject conditions (e.g., brightness, contrast, frequency characteristics, etc.), and the shooting conditions (e.g., exposure time, aperture, sensitivity, etc.).

[0190] That is, for example, the third defective pixel correction method (for example, the operation of the calculation unit performed in step S1708) may be controlled according to at least one of the type of defect, the subject condition, and the shooting condition. Such control is also included in this embodiment.

[0191] The present invention has been described above in detail based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible based on the spirit of the present invention, and these are not excluded from the scope of the present invention.

[0192] The present invention also includes those that realize the functions of the above embodiments using, for example, at least one processor such as a CPU, memory, or circuit (for example, ASIC). Also, multiple processors may be used to perform distributed processing.

[0193] In order to realize some or all of the control in the above-described embodiments, a computer program that realizes the functions of the above-described embodiments may be supplied to an imaging device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the imaging device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. The present invention also includes the following combinations.

[0194] (Configuration 1) An imaging device comprising: an imaging element having a plurality of pixels, each pixel having a photoelectric conversion unit, a first charge holding unit for holding the output of the photoelectric conversion unit, and a second charge holding unit for holding the output of the photoelectric conversion unit; and a defective pixel correction unit that, when the first charge holding unit of a specified pixel is defective, calculates an interpolated pixel value for interpolating the pixel value from the first charge holding unit based on the pixel value output from the second charge holding unit of the pixel.

[0195] (Configuration 2) The imaging device described in Configuration 1, characterized in that the pixel has a first charge transfer unit that transfers charges from the photoelectric conversion unit to the first charge storage unit, a second charge transfer unit that transfers charges from the photoelectric conversion unit to the second charge storage unit, and a floating diffusion unit to which charges from the first charge transfer unit and the second charge storage unit are transferred.

[0196] (Configuration 3) An imaging device according to configuration 1 or 2, characterized in that the charge accumulated in the photoelectric conversion unit for a first exposure time is held in the first charge holding unit, and the charge accumulated in the photoelectric conversion unit for a second exposure time shorter than the first exposure time is held in the second charge holding unit.

[0197] (Configuration 4) The imaging device described in Configuration 3, characterized in that the defective pixel correction unit calculates the interpolated pixel value based on the pixel value read out from the second charge storage unit in accordance with the ratio between the first exposure time and the second exposure time.

[0198] (Configuration 5) An imaging device described in any one of configurations 1 to 4, characterized in that when a defect is detected in the second charge storage unit, the charge accumulated in the photoelectric conversion unit for a first exposure time is stored in the second charge storage unit, and the charge accumulated in the photoelectric conversion unit for a second exposure time shorter than the first exposure time is stored in the first charge storage unit.

[0199] (Configuration 6) An imaging device described in any one of configurations 1 to 5, characterized in that when the number of defects in the second charge holding unit of a plurality of the pixels is greater than the number of defects in the first charge holding unit, in the plurality of the pixels, the charge accumulated in the photoelectric conversion unit for the first exposure time is held in the second charge holding unit, and the charge accumulated in the photoelectric conversion unit for a second exposure time shorter than the first exposure time is held in the first charge holding unit.

[0200] (Configuration 7) An imaging device according to any one of configurations 1 to 6, characterized in that it has a synthesis processing unit that synthesizes signals read out from the imaging element to generate a high dynamic range image, and that the defective pixel correction unit performs defective pixel correction processing prior to the synthesis processing by the synthesis processing unit.

[0201] (Configuration 8) An imaging device described in any one of configurations 1 to 7, characterized in that the defective pixel correction unit has a calculation unit that calculates the interpolated pixel value for interpolating the pixel value from the first charge storage unit based on the pixel values ​​of pixels surrounding the specified pixel when the first charge storage unit of the specified pixel is defective.

[0202] (Configuration 9) The imaging device described in Configuration 8 is characterized in that, when there is a defect in the first charge storage unit of a specified pixel, the defective pixel correction unit calculates the interpolated pixel value for interpolating the pixel value from the first charge storage unit based on the pixel value output from the second charge storage unit of the pixel and the pixel values ​​of pixels surrounding the specified pixel.

[0203] (Configuration 10) The imaging device according to configuration 8 or 9, wherein pixel values ​​of the surrounding pixels that are equal to or greater than a predetermined value are not used in calculating the interpolated pixel value.

[0204] (Configuration 11) The imaging device according to any one of configurations 8 to 10, wherein the operation of the calculation unit is controlled in accordance with at least one of the type of the defect, the condition of the subject, and the imaging condition.

[0205] (Structure 12) An imaging device comprising: an imaging element having a plurality of pixels, each pixel having a first photoelectric conversion unit, a second photoelectric conversion unit, a first charge holding unit for holding the output from the first photoelectric conversion unit and the second photoelectric conversion unit, respectively, and a second charge holding unit for holding the output from the first photoelectric conversion unit and the second photoelectric conversion unit, respectively; and a defective pixel correction unit that, when there is a defect in the first charge holding unit for holding the output of the first photoelectric conversion unit of a specific pixel, calculates an interpolated pixel value for interpolating the pixel value from the first charge holding unit for holding the output of the first photoelectric conversion unit based on the pixel value output from the first charge holding unit for holding the output of the second photoelectric conversion unit of the pixel.

[0206] (Method) A processing method for processing signals from an imaging element having multiple pixels, each pixel having a photoelectric conversion unit, a first charge holding unit for holding the output of the photoelectric conversion unit, and a second charge holding unit for holding the output of the photoelectric conversion unit, characterized in that when there is a defect in the first charge holding unit of a specified pixel, the processing method includes a defective pixel correction step of calculating an interpolated pixel value for interpolating the pixel value from the first charge holding unit based on the pixel value output from the second charge holding unit of the pixel.

[0207] (Program) A computer program for controlling each unit of the imaging device according to any one of configurations 1 to 12 by a computer. [Explanation of symbols]

[0208] 101: Pixel section 102: Vertical scanning circuit 103: Column amplifier circuit 104: Horizontal scanning circuit 105: Output circuit 106: Control circuit 107: Pixel 108: Column signal line 217: Defective pixel correction unit 100: Image sensor 200: Imaging device

Claims

1. a photoelectric conversion unit; a first charge holding unit for holding an output of the photoelectric conversion unit; a second charge holding unit for holding an output of the photoelectric conversion unit; an imaging element having a plurality of pixels each having and a defective pixel correction unit that, when the first charge storage unit of a specified pixel has a defect, calculates an interpolated pixel value for interpolating the pixel value from the first charge storage unit based on the pixel value output from the second charge storage unit of the pixel.

2. The pixel is a first charge transfer unit that transfers charges from the photoelectric conversion unit to the first charge holding unit; a second charge transfer unit that transfers charges from the photoelectric conversion unit to the second charge holding unit; 2. The imaging device according to claim 1, further comprising: the first charge transfer section; and a floating diffusion section to which the charges from the second charge holding section are transferred.

3. 2. The imaging device according to claim 1, wherein charges accumulated in the photoelectric conversion unit for a first exposure time are held in the first charge holding unit, and charges accumulated in the photoelectric conversion unit for a second exposure time shorter than the first exposure time are held in the second charge holding unit.

4. 4. The imaging device according to claim 3, wherein the defective pixel correction unit calculates the interpolated pixel value based on the pixel value read from the second charge storage unit in accordance with a ratio between the first exposure time and the second exposure time.

5. 2. The imaging device according to claim 1, wherein, when a defect is detected in the second charge storage unit, the charge accumulated in the photoelectric conversion unit for a first exposure time is stored in the second charge storage unit, and the charge accumulated in the photoelectric conversion unit for a second exposure time shorter than the first exposure time is stored in the first charge storage unit.

6. 2. The imaging device according to claim 1, wherein, when the number of defects in the second charge storage units of the plurality of pixels is greater than the number of defects in the first charge storage units, in the plurality of pixels, the charges accumulated in the photoelectric conversion units for the first exposure time are stored in the second charge storage units, and the charges accumulated in the photoelectric conversion units for a second exposure time shorter than the first exposure time are stored in the first charge storage units.

7. The imaging device according to claim 1, further comprising a synthesis processing unit that synthesizes signals read from the imaging element to generate a high dynamic range image, and wherein the defective pixel correction unit performs defective pixel correction processing prior to the synthesis processing by the synthesis processing unit.

8. 2. The imaging device according to claim 1, wherein the defective pixel correction unit includes a calculation unit that, when the first charge storage unit of the specified pixel has a defect, calculates the interpolated pixel value for interpolating the pixel value from the first charge storage unit based on pixel values ​​of pixels surrounding the specified pixel.

9. 9. The imaging device according to claim 8, wherein, when the first charge storage unit of a specific pixel has a defect, the defective pixel correction unit calculates the interpolated pixel value for interpolating the pixel value from the first charge storage unit based on the pixel value output from the second charge storage unit of the specific pixel and the pixel values ​​of pixels surrounding the specific pixel.

10. 9. The imaging device according to claim 8, wherein pixel values ​​of the surrounding pixels that are equal to or greater than a predetermined value are not used in calculating the interpolated pixel value.

11. 9. The imaging apparatus according to claim 8, wherein the operation of the calculation unit is controlled in accordance with at least one of the type of the defect, the condition of the subject, and the imaging condition.

12. a first photoelectric conversion unit; a second photoelectric conversion unit; a first charge holding unit for holding outputs from the first photoelectric conversion unit and the second photoelectric conversion unit, respectively; a second charge holding unit for holding outputs from the first photoelectric conversion unit and the second photoelectric conversion unit, respectively; an imaging element having a plurality of pixels each having and a defective pixel correction unit that, when there is a defect in the first charge holding unit for holding the output of the first photoelectric conversion unit of a specified pixel, calculates an interpolated pixel value for interpolating the pixel value from the first charge holding unit for holding the output of the first photoelectric conversion unit, based on the pixel value output from the first charge holding unit for holding the output of the second photoelectric conversion unit of the pixel.

13. a photoelectric conversion unit; a first charge holding unit for holding an output of the photoelectric conversion unit; a second charge holding unit for holding an output of the photoelectric conversion unit; A processing method for processing a signal from an image sensor having a plurality of pixels each having A processing method characterized by comprising a defective pixel correction step of, when a defect exists in the first charge storage unit of a predetermined pixel, calculating an interpolated pixel value for interpolating a pixel value from the first charge storage unit based on a pixel value output from the second charge storage unit of the pixel.

14. A computer program for controlling each unit of the imaging device according to any one of claims 1 to 12 by a computer.

Images