Imaging apparatus and method of operating an imaging apparatus

Abstract

This application relates to an imaging apparatus and a method of operating the imaging apparatus. An imaging apparatus is disclosed, comprising: an image sensing unit including a first pixel and a second pixel having image data smaller than that of the first pixel; and an image signal processor (ISP) configured to perform image processing on first image data generated from the first pixel and second image data generated from the second pixel, wherein the ISP can be configured to synthesize the first image data and the second image data based on pixel information per coordinate, and the light intensity used to saturate the second image data of the second pixel can be greater than the light intensity configured to include the first imaging data of the first pixel.

Description

Technical Field

[0001] Embodiments of this disclosure relate to an imaging apparatus and a method of operating the imaging apparatus. Background Technology

[0002] With the development of the information and communication industry and the digitization of electronic devices, image sensors with improved performance are being used in various fields such as digital cameras, camcorders, mobile phones, PCS (personal communication systems), gaming devices, security cameras, and medical miniature cameras. Typically, an image sensor has a pixel area comprising a photodiode and peripheral circuitry. A single pixel includes a photodiode and a transistor. The transistor is placed between the photodiode and a floating diffusion region to transfer the charge generated by the photodiode to the floating diffusion region. Summary of the Invention

[0003] Various implementations of the disclosed technology provide an imaging device with high dynamic range by synthesizing image data generated from pixels with different image data sizes.

[0004] Various implementations of the disclosed technology provide a method for operating an imaging device that has a high dynamic range by synthesizing image data generated from pixels of image data with different sizes.

[0005] In one aspect, an imaging apparatus may include: an image sensing unit including a first pixel and a second pixel having image data smaller than that of the first pixel; and an ISP (image signal processor) configured to perform image processing on first image data generated from the first pixel and second image data generated from the second pixel, wherein the ISP may be configured to synthesize the first image data and the second image data based on pixel information per coordinate, and to saturate the second image data of the second pixel with a light intensity greater than the light intensity of the first imaging data of the first pixel configured to include the first pixel.

[0006] In another aspect of this disclosure, an imaging apparatus according to another embodiment may include: a pixel array including a first pixel and a second pixel having image data smaller than that of the first pixel; and an ISP for processing first image data generated from the first pixel and second image data generated from the second pixel as image signals, wherein the first pixel and the second pixel each include a photoelectric conversion element, a transmission transistor electrically connected to the photoelectric conversion element, and a floating diffusion region connected to the transmission transistor.

[0007] A method of operating an imaging apparatus according to another embodiment may include the following steps: capturing a pixel image of a plurality of pixels in a wafer test; extracting image data of each of the plurality of pixels; generating information about pixels distinguished by coordinates by distinguishing the plurality of pixels based on the image data of the plurality of pixels; and generating an HDR (High Dynamic Range) image based on the information about pixels distinguished by coordinates.

[0008] According to embodiments of this disclosure, by synthesizing image data generated from pixels with different image data sizes, the imaging device can have a high dynamic range.

[0009] However, it should be understood that although the detailed description and specific examples illustrate preferred embodiments of the invention, they are for illustrative purposes only, as those skilled in the art can clearly understand various changes and modifications within the spirit and scope of the invention from the detailed description. Attached Figure Description

[0010] Figure 1 This is an example diagram of an imaging system based on some implementations of the disclosed technology.

[0011] Figure 2 It is based on some implementation methods of the disclosed technology. Figure 1 An example diagram of an image sensing unit is shown.

[0012] Figure 3 It is based on some implementation methods of the disclosed technology. Figure 2 An example of a circuit diagram for the pixels of the pixel array shown.

[0013] Figure 4 This is an example of a graph showing the potential of a bad pixel in each time period based on some implementations of the disclosed technology.

[0014] Figure 5 This is a view showing image data output from pixels based on some implementations of the disclosed technology.

[0015] Figure 6 This is an example of a graph showing the number of image data pixels for each of various pixel array samples based on some implementations of the disclosed technology.

[0016] Figure 7 This is an example of a graph showing the number of pixels in an image data sample of a pixel array based on some implementations of the disclosed technology.

[0017] Figure 8 This is an example of a view showing image data for each of the first, second, and third pixels based on some implementations of the disclosed technology.

[0018] Figure 9 This is an example of a graph showing image data over time based on some implementations of the disclosed technology.

[0019] Figure 10 This is an example of a graph showing the sensitivity of the first, second, and third pixels based on some implementations of the disclosed technology.

[0020] Figure 11 This is an example of a graph showing the conversion gain based on some implementations of the disclosed technology.

[0021] Figure 12 This is an example of a view showing how the duration of light illumination is controlled based on some implementations of the disclosed technology.

[0022] Figure 13 This illustrates how, based on some implementations of the disclosed technology, a high dynamic range curve is generated from image data of each of the first, second, and third pixels of a synthetic imaging device.

[0023] Figure 14 This is an example of a flowchart illustrating a method for operating an imaging device based on some implementations of the disclosed technology. Detailed Implementation

[0024] A detailed description will now be given with reference to the accompanying drawings and the exemplary embodiments disclosed herein.

[0025] For the purpose of brief description with reference to the accompanying drawings, the same or equivalent components may be given the same reference numerals and their descriptions will not be repeated.

[0026] These terms are generally used only to distinguish one element from another. It should be understood that the terms "first" and "second" are used herein to describe various components, but these components should not be limited by these terms. The terms above are used only to distinguish one component from another. For example, a first component may be referred to as a second component without departing from the scope of the invention, and vice versa. Unless the context clearly indicates otherwise, singular expressions include plural expressions.

[0027] The terminology used herein includes terms such as “comprising” or “including”, and it should be understood that these terms are intended to indicate the presence of several components, functions, or steps disclosed in the specification, and that more or fewer components, functions, or steps may also be used. However, this disclosure may be embodied in various modified examples and is not limited to the embodiments described herein.

[0028] Figure 1 This is a block diagram illustrating an imaging system according to one embodiment.

[0029] Reference Figure 1 Imaging system 1 can refer to a device such as a digital still camera for capturing still images or a digital video camera for imaging moving images, as well as a device for detecting motion. For example, imaging device 10 can be implemented as a digital SLR (DSLR) camera, a mirrorless camera, or a mobile phone (especially a smartphone), but the implementation is not limited to these. Imaging device 10 can be a concept that includes a means of imaging an object and generating an image by means of a lens and an imaging element.

[0030] The imaging system 1 may include an imaging device 10 and a host device 20.

[0031] The imaging device 10 may include an image sensing unit 100, a line memory 200, an image signal processor (ISP) 300, an input / output interface (I / O interface) 400, and a data memory 500.

[0032] In this example, the image sensing unit 100 may be a CIS (Complementary Metal-Oxide-Semiconductor Image Sensor) configured to convert optical signals into electrical signals. The overall operation of the image sensing unit 100, including on / off states, operating modes, operating timing, and sensitivity, can be controlled by the ISP 300. The sensitivity of a pixel in the image sensing unit 100 refers to its ability to convert incident light into an electrical signal. In this example, the increase in image data within a pixel during exposure time is based on the pixel's sensitivity. The image sensing unit 100 may convert optical signals into electrical signals under the control of the ISP 300 and send the image data to the line memory 200.

[0033] Line memory 200 may include volatile memory (e.g., DRAM, SRAM) and / or non-volatile memory (e.g., flash memory).

[0034] The line memory 200 can receive and store image data from the image sensing unit 100, and send the stored image data to the ISP 300 under the control of the ISP 300.

[0035] The ISP 300 can perform image signal processing on image data stored in the online memory 200. To reduce noise and improve image quality, the ISP 300 can perform image signal processing on the image data, such as gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, and lens distortion correction.

[0036] To generate HDR images, the ISP 300 may include a gain processing unit 310 and an image synthesis unit 320.

[0037] The gain processing unit 310 can determine the gain to be applied to the image data (e.g., multiplication).

[0038] The image synthesis unit 320 can use image data generated from each of the first pixel PXa, the second pixel PXb, and the third pixel PXc to synthesize an HDR image corresponding to the high dynamic range.

[0039] In one implementation, the data memory 500 of the imaging apparatus 10, according to one embodiment, can transmit coordinate-based pixel information to the ISP 300. The coordinate-based pixel information can refer to information associated with each pixel indexed according to its specific location (coordinates). The coordinate-based pixel information may include pixel identification information indicating whether the corresponding pixel belongs to a first pixel (PXa) group, a second pixel (PXb) group, or a third pixel (PXc) group. In one implementation, the coordinate-based pixel information is generated immediately after wafer testing and stored in the data memory 500. However, the implementation of the disclosed technology is not limited to this. For example, in another implementation, the coordinate-based pixel information can be generated when the ISP 300 performs image processing on a captured object in response to the object being captured by the imaging apparatus 10.

[0040] The pixel information by coordinates will be described in more detail later.

[0041] The ISP 300 can send image data (i.e., HDR images) processed by image signal processing to the input / output interface 400.

[0042] According to another embodiment, the gain processing unit 310 and the image synthesis unit 320 for generating HDR images may be included in the image sensing unit 100 instead of the ISP 300.

[0043] The input / output interface 400 communicates with the host device 20 and can send image data processed by image signal processing to the host device 20.

[0044] The host device 20 may be a processor (e.g., an application processor) that processes image data obtained from the video signal received from the imaging device 10, a memory (e.g., a non-volatile memory) that stores the image data, or a display device (e.g., a liquid crystal display, LCD) that visually outputs the image data.

[0045] The data storage device 500 can store image data in digital format, Data_PX, converted from analog-to-digital converter 140.

[0046] Figure 2 It is shown in detail Figure 1 The image sensing unit shown is a view.

[0047] Reference Figure 2 The image sensing unit 100 may include a pixel array 110, a row driver 120, a correlated dual sampler (CDS) 130, an analog-to-digital converter (ADC) 140, an output buffer 150, a column driver 160, and a timing controller 170. Each configuration of the image sensing unit 100 is merely exemplary, and at least some configurations may be added or omitted as needed.

[0048] Pixel array 110 may include a plurality of image pixels arranged in multiple rows and columns. In one embodiment, the plurality of image pixels may be arranged in a two-dimensional pixel array including rows and columns. In another embodiment, the plurality of image pixels may be arranged in a three-dimensional pixel array. The plurality of image pixels may convert optical signals into electrical signals on a pixel-by-pixel or pixel-by-group basis, and image pixels within a pixel-by-pixel group may share at least some internal circuitry. Pixel array 110 may receive pixel control signals, including a row selection signal, a pixel reset signal, and a transmission signal, from row driver 120, and corresponding pixels of pixel array 110 may be activated by the pixel control signals to perform operations corresponding to the row selection signal, pixel reset signal, and transmission signal. Each of the image pixels generates a photocharge corresponding to the intensity (or illuminance) of the incident light and generates an electrical signal having a magnitude corresponding to the amount of photocharge generated, thereby detecting the incident light. For ease of explanation, image pixel may also be referred to as pixel.

[0049] The row driver 120 can activate the pixel array 110 based on commands and control signals provided by the timing controller 170 to perform specific operations on the pixels included in the row.

[0050] In one implementation, the correlated double sampler 130 can sequentially sample and hold reference signals and image signals provided from the pixel array 110 to each of the multiple column lines. That is, the correlated double sampler 130 can sample and hold the levels of the reference signals and image signals corresponding to each column of the pixel array 110.

[0051] The correlated double sampler 130 can send the reference signal and image signal of each column as correlated double sample signals to the ADC 140 based on the control signal from the timing controller 170.

[0052] The ADC 140 can convert the correlated double sampled signal of each column output from the correlated double sampler 130 into a digital signal and output image data. In one embodiment, the ADC 140 can convert the correlated double sampled signal generated by the correlated double sampler 130 for each column into a digital signal and output the digital signal.

[0053] The ADC 140 may include multiple column counters corresponding to each column of the pixel array 110. Each column of the pixel array 110 is connected to each column counter, and image data can be generated by converting the associated double-sampled signal corresponding to each column into a digital signal using the column counters.

[0054] Output buffer 150 can temporarily hold and output image data for each column provided by ADC 140. Output buffer 150 can temporarily store image data output from ADC 140 based on control signals from timing controller 170.

[0055] The column driver 160 can select the column of the output buffer 150 based on the control signal of the timing controller 170, and control the output buffer 150 so that the image data temporarily stored in the selected column of the output buffer 150 is output sequentially.

[0056] The timing controller 170 can control at least one of the row driver 120, the correlated dual sampler 130, the ADC 140, the output buffer 150, and the column driver 160.

[0057] The timing controller 170 can provide at least one of the row driver 120, the correlated dual sampler 130, the ADC 140, the output buffer 150, and the column driver 160 with clock signals required for the operation of each component of the image sensing unit 100, control signals for timing control, and address signals for selecting rows or columns. According to one embodiment, the timing controller 170 may include logic control circuitry, a phase-locked loop (PLL) circuit, timing control circuitry, and communication interface circuitry.

[0058] Figure 3 yes Figure 2 The circuit diagram of the pixels in the pixel array shown.

[0059] Reference Figure 3 The circuit diagram of a pixel can correspond to a pixel array (see...). Figure 2 The equivalent circuit of each pixel PX included in 110).

[0060] A pixel PX may include a photoelectric conversion element PD, a transmission transistor TT, a reset transistor RT, a floating diffusion region FA, a drive transistor DT, and a selection transistor ST. That is, a pixel may have a 4TR (4 transistors) pixel structure.

[0061] A photoelectric conversion element (PD) can generate and accumulate photocharge corresponding to the intensity of incident light. For example, a photoelectric conversion element (PD) can be implemented as a photodiode, a phototransistor, a photogate, a pinned photodiode, or a combination thereof. When a photoelectric conversion element (PD) is implemented as a photodiode, it can be a region within a substrate having a first conductivity type (e.g., P-type) doped with impurities of a second conductivity type (e.g., N-type).

[0062] A transfer transistor TT can be connected between the photoelectric conversion element PD and the floating diffusion region FA. The transfer transistor TT can be turned on or off in response to a transfer control signal TX applied to the gate electrode TG, and when turned on, the transfer transistor TT can transfer the photocharge accumulated in the photoelectric conversion element PD to the floating diffusion region FA. The source electrode of the transfer transistor TT can be connected to the photoelectric conversion element PD, and the drain electrode can be connected to the floating diffusion region FA.

[0063] A reset transistor RT is connected between the power supply voltage VDD and the floating diffusion region FA, and can reset the voltage of the floating diffusion region FA to the power supply voltage VDD in response to a reset control signal RX applied to the gate electrode RG. The power supply voltage VDD can be applied to the source electrode of the reset transistor RT, and the drain electrode can be connected to the floating diffusion region FA.

[0064] The floating diffusion region FA can accumulate photocharge transferred from the transfer transistor TT. For example, the floating diffusion region FA can be a region within a substrate having a first conductivity type (e.g., P-type) doped with an impurity of a second conductivity type (e.g., N-type), and the substrate and the impurity-doped region can be modeled as a junction capacitor. The floating diffusion region FA can be connected to the gate electrode of the driving transistor DT, the drain electrode of the transfer transistor TT, and the drain electrode of the reset transistor RT. Floating diffusion electrodes can be arranged within the floating diffusion region FA.

[0065] The driving transistor DT is connected between the power supply voltage VDD and the selection transistor ST. It amplifies the potential change of the floating diffusion region FA, which receives the accumulated photocharge in the photoelectric conversion element PD, and transmits this amplified potential change to the selection transistor ST. The gate electrode of the driving transistor DT is connected to the floating diffusion region FA. The power supply voltage VDD is applied to the source electrode (or drain electrode), and the drain electrode (or source electrode) can be connected to the source electrode of the selection transistor ST.

[0066] The select transistor ST is connected between the drive transistor DT and the output signal line (or column line), and is turned on by the row select control signal SX applied to the gate electrode SG to output the electrical signal transmitted from the drive transistor DT as the pixel signal Vout. The source electrode of the select transistor ST can be connected to the drain electrode (or source electrode) of the drive transistor DT.

[0067] The voltage of the floating diffusion region FA can be determined by the amount of photocharge transmitted via the transmission transistor TT, and the voltage of the floating diffusion region FA can decrease as the amount of photocharge increases.

[0068] The floating diffusion region FA (or floating diffusion electrode) can form a capacitor with the boost control line FDBL.

[0069] Each of the control signal, transmission control signal TX, reset control signal RX, and row selection control signal SX provided to the boost control line FDBL can be provided from the row driver 120, but the implementation is not limited thereto.

[0070] According to this specification, a pixel PX may include a first time period T1 to a fourth time period T4. In the following text, for ease of explanation, the reset time period is referred to as the first time period T1, the illumination time period as the second time period T2, and the readout time period (lead period) as the third time period T3 and the fourth time period T4. The third time period T3 may be the FD reset time period, and the fourth time period T4 may be the lead-out time period. The first time period T1 may be the PD reset time period.

[0071] The potential changes of cold pixels (or bad pixels) from the first time period T1 to the fourth time period T4, as described above, will be further described below. Cold pixels or bad pixels refer to defective pixels.

[0072] Figure 4 It is a view showing the potential of bad pixels in each time period.

[0073] Reference Figure 4 During the first time period T1, the photocharge of the photoelectric conversion element PD can be completely reset.

[0074] During the second time period T2, photocharge (signal) can accumulate in the photoelectric conversion element PD.

[0075] In the fourth time period T4, when the transmission transistor TT is turned on, the photocharge (signal) accumulated in the photoelectric conversion element PD in the second time period T2 can move to the floating diffusion region FA. However, due to... Figure 8The pixels are cold pixels (ColdPX) (or bad pixels), so only some photocharge (Signal_O) accumulated in the photoelectric conversion element PD can accumulate in the floating diffusion region FA, and the remaining photocharge (Signal_P) may not accumulate in the photoelectric conversion element PD. Therefore, when each of the cold pixels and normal pixels receives the same light, the image data generated from the cold pixels (Cold PX) can be less than the image data generated from the normal pixels (Normal PX).

[0076] For example, a cold pixel (Cold PX) could be a pixel in which some photocharge (signal) accumulated in the photoelectric conversion element (PD) leaks to the transmission transistor (TT) due to a defect in the transmission transistor (TT) during the fourth time period (T4). However, other implementations are possible, and the implementation is not limited to this.

[0077] In the case of a normal pixel, the same amount of photocharge (signal) can accumulate in the floating diffusion region FA as in the photoelectric conversion element PD.

[0078] Figure 9 This is a view showing how image data output from pixels is stored according to one embodiment.

[0079] Figure 5 The attached diagram explains how pixel signals S1_PX and S2_PX, extracted (or generated) from the third time period T3 and the fourth time period T4 of pixel PX, are stored in the data memory 500.

[0080] Reference Figure 5 The first pixel signal S1_PX can be output to the output signal line in the third time period T3, while the second pixel signal S2_PX can be output to the output signal line in the fourth time period T4. The output pixel signals S1_PX and S2_PX are sent to the correlated dual sampler 130.

[0081] This can be performed during the wafer testing phase. Figure 5 The pixel signals S1_PX and S2_PX extracted (or generated) from the third time period T3 and the fourth time period T4 of the pixel PX shown are processed and stored in the data memory 500.

[0082] In some implementations, the generation of pixel signals S1_PX and S2_PX can be performed in a white environment. A white environment refers to a condition where the image sensing unit is exposed to a uniform, bright, and typically spectrally neutral (white) light source. However, this implementation is not limited to this. In other implementations, the generation of pixel signals S1_PX and S2_PX can be performed in a black environment. A black environment refers to a controlled test or operating condition where the image sensing unit is placed in complete or near-complete darkness. When the generation of pixel signals S1_PX and S2_PX is performed in a black environment, there may be no accumulated photocharge in the photoelectric conversion element PD during the second time period T2.

[0083] For example, in the case of a normal pixel (Normal PX), the first pixel signal S1_PX can be a first voltage value, and the second pixel signal S2_PX can be a second voltage value. The difference between the first voltage value and the second voltage value can correspond to the amount of photocharge applied to the photoelectric conversion element (PD).

[0084] The correlated double sampler 130 can calculate the difference between the transmitted pixel signals S1_PX and S2_PX and provide this difference to the ADC 140. Therefore, the ADC 140 can receive a voltage value corresponding to the difference between the first and second voltage values ​​and perform digital conversion and offset. For example, the ADC 140 can perform digital conversion and offset on the voltage value corresponding to the difference between the first and second voltage values ​​provided from the correlated double sampler 130 to generate image data Data_PX (Pedestal[LSB]), whose value can be 64[LSB]. For example, assume that the image data[LSB] generated from normal pixels is set to 64[LSB]. LSB can represent "least significant bit," and this LSB can be used as a unit of digital output for pixels.

[0085] However, in the case of cold pixels (Cold PX), the first pixel signal S1_PX may have a first voltage value, and the second pixel signal S2_PX may have a voltage lower than the second voltage value (e.g., a third voltage value). The correlated dual sampler 130 calculates the difference between the transmitted pixel signals (S1_PX, S2_PX) and provides this difference to the ADC 140. Therefore, the ADC 140 can receive the voltage value corresponding to the difference between the first and third voltage values ​​and perform digital conversion and offset. For example, the ADC 140 can perform digital conversion and offset on the voltage value corresponding to the difference between the first and second voltage values ​​provided from the correlated dual sampler 130 to generate image data Data_PX (Pedestal[LSB]), whose value may be 64[LSB]. For example, it is assumed that the image data[LSB] generated from normal pixels is set to 64[LSB].

[0086] According to this embodiment, as described above, the size of the image data Data_PX generated from each pixel PX differs based on normal pixels and cold pixels, and the image data Data_PX generated for each pixel PX can be stored in the data memory 500. The data memory 500 can store the image data Data_PX so that it corresponds to the pixel PX at each coordinate.

[0087] Figure 6 It is a graph showing the number of image data pixels for each of the various pixel array samples.

[0088] Figure 6 The above is shown in the first pixel array sample #1, the second pixel array sample #2, and the third pixel array sample #3. Figure 5 The image data Data_PX value generated for each pixel PX in the image, and the number of pixels PX corresponding to each value.

[0089] Reference Figure 6 For the first pixel array sample #1, the image data Data_PX of most pixels PX can be greater than 50 [LSB], and for the second pixel array sample #2 and the third pixel array sample #3, the image data Data_PX of multiple pixels PX is between 0 and 50 [LSB].

[0090] For the first pixel array sample #1, there are not many pixels PX with less than 50 [LSB], so the first pixel array sample #1 may not contain many cold pixels.

[0091] On the other hand, for the first pixel array sample #2 and the second pixel array sample #3, there are some pixel PX with less than 50 [LSB], so they can contain a large number of cold pixels.

[0092] However, cold pixels may not have precise numerical values ​​or standards and can be determined based on [LSB] as needed.

[0093] Furthermore, by analyzing the image data Data_PX generated for each pixel PX of these pixel array samples #1 to #3, it is possible to distinguish pixel arrays capable of generating HDR images corresponding to high dynamic range by synthesizing image data generated from each of the normal and cold pixels. That is, the first pixel array sample #1 may have very few cold pixels, in which case the first pixel array sample #1 may not be suitable for generating HDR images corresponding to high dynamic range by synthesizing image data generated from each of the normal and cold pixels.

[0094] Below, we will explain how to distinguish between cold pixels and normal pixels based on, for example, the third pixel array sample #3.

[0095] Figure 7 It is a graph showing the number of pixels in the image data generated from the pixels of a pixel array sample.

[0096] Reference Figure 7 For the third pixel array sample #3 (or a pixel array according to one embodiment), the image data generated from the pixels in the third pixel array sample #3 can be divided into multiple segments with different brightness levels. In such cases... Figure 7 In the example shown, the image data generated from the pixels in the third pixel array sample #3 can be divided into three segments: a first image data Data_PXa of approximately 50 [LSB] or greater, a second image data Data_PXb between approximately 30 [LSB] and approximately 50 [LSB], and a third image data Data_PXc of less than approximately 30 [LSB]. However, [the text abruptly ends here, likely due to an incomplete sentence or a formatting error]. Figure 7 The image data is divided into three segments, and the values ​​are not limited to these, and can be changed in various ways as needed.

[0097] Figure 8 It is a view that shows the image data of each of the first, second, and third pixels.

[0098] Reference Figure 8 , has in Figure 7 The pixels of the first image data Data_PXa are defined as first pixel PXa, the pixels of the second image data Data_PXb are defined as second pixel PXb, and the pixels of the third image data Data_PXc are defined as third pixel PXc.

[0099] Figure 8 An example of the arrangement of the first pixel PXa, the second pixel PXb, and the third pixel PXc of the image sensing unit 100 is shown. However, Figure 8 The arrangement and number of pixels PXa, PXb and PXc shown are not limited to this.

[0100] According to one embodiment, the image sensing unit 100 can generate an image with high dynamic range (HDR) by synthesizing image data generated from pixels PXa, PXb, and PXc that have different image data Data_PXa, Data_PXb, and Data_PXc.

[0101] Figure 9 It is a graph showing image data over time according to one embodiment.

[0102] Reference Figure 9The diagram illustrates the variation of image data Data_PXa, Data_PXb, and Data_PXc generated from each of the first pixel PXa, the second pixel PXb, and the third pixel PXc according to the illumination time Int.time [LSB].

[0103] like Figure 9 As shown, based on the illumination time Int.time, the changes in image data Data_PXa, Data_PXb, and Data_PXc generated from each of the first pixel PXa, the second pixel PXb, and the third pixel PXc can be the same. That is, the sensitivities of the first pixel PXa, the second pixel PXb, and the third pixel PXc can be the same.

[0104] Figure 10 It is a graph showing the sensitivity of the first pixel, the second pixel, and the third pixel according to one embodiment.

[0105] Reference Figure 10 ,like Figure 9 As shown, the sensitivity of the first pixel PXa, the second pixel PXb, and the third pixel PXc can be the same.

[0106] The reason why the first pixel PXa, the second pixel PXb, and the third pixel PXc have the same sensitivity will be explained below.

[0107] As described above, when each of the cold and normal pixels receives the same light, the image data generated from the cold pixel (ColdPX) can be smaller than the image data generated from the normal pixel (NormalPX). For example, in the cold pixel (ColdPX), due to defects in the transmission transistor, some of the photocharge (signal) accumulated in the photoelectric conversion element (PD) can leak into the transmission transistor during the fourth time period (T4). Here, the photocharge leaking into the transmission transistor can be fixed for each cold pixel, and the size of the generated image data (Data_PX) varies based on the photocharge leakage. That is, in the case of the cold pixel, besides the photocharge leaking into the transmission transistor, the remaining photocharge (signal) moves to the floating diffusion region (FA).

[0108] Therefore, in Figure 10 In the case of the first pixel PXa, the second pixel PXb, and the third pixel PXc, only the sizes of the initial image data Data_PXa, Data_PXb, and Data_PXc are different, while the sensitivity can be the same.

[0109] An imaging apparatus according to one embodiment (see [see embodiment]) Figure 1 10) data storage (see Figure 1 (500) can send pixel information by coordinates to the ISP (see 500) Figure 1(300). Pixel information by coordinates can be generated immediately after the wafer testing is completed, but the implementation is not limited to this. It can also be generated when the user uses an imaging device (see...). Figure 1 10) The subject being photographed and the ISP (see Figure 1 (300) is generated when the image of the photographed object is processed.

[0110] Pixel information by coordinates can be information about classifying pixels (PX) by coordinates (first pixel PXa, second pixel PXb, and third pixel PXc).

[0111] Figure 11 This is an example of a graph illustrating how to generate a high dynamic range image by adjusting the conversion gain. Figure 12 This is an example of a graph showing the adjustment of exposure time.

[0112] Figure 11 The horizontal axis represents light intensity, and the vertical axis represents image data Data_PX[LSB]. Figure 12 The horizontal axis can represent brightness (or light intensity).

[0113] Figure 11 and Figure 12 It demonstrates that high dynamic range images can be generated by controlling the conversion gain and exposure time together.

[0114] Reference Figure 11 For example, a mode with three conversion gains CG1, CG2, and CG3 is shown. The first conversion gain CG1 can be the largest, the third conversion gain CG3 can be the smallest, and the second conversion gain CG2 can be between the first conversion gain CG1 and the third conversion gain CG3.

[0115] Gain processing unit (see) Figure 1 (310 in the text) can determine the gain based on the difference between different conversion gains CG1, CG2 and CG3, and provide that gain to the image synthesis unit (see 310 in the text). Figure 1 (320 in the middle). The gain based on the difference in conversion gain can be determined in advance through experiments and stored in the gain processing unit 310.

[0116] It can be said that the slope of the image data relative to illuminance (or light intensity) is different in the first conversion gain CG1 mode, the second conversion gain CG2 mode, and the third conversion gain CG3 mode.

[0117] The first conversion gain CG1 mode can operate in the third exposure environment t3 with the longest exposure time t3, the second conversion gain CG2 mode can operate in the second exposure environment t2 with an intermediate exposure time t2 shorter than t3, and the third conversion gain CG3 mode can operate in the first exposure environment t1 with the shortest exposure time t1 less than t2.

[0118] Furthermore, the imaging device can take multiple shots, such that the exposure time is the shortest in the first exposure environment t1, the second shortest or in the middle in the second exposure environment t2, and the longest in the third exposure environment t3.

[0119] Assume the image object comprises three parts with varying brightness: the brightest part, a mid-tone or second-brightest part, and the darkest part. Since the first exposure environment t1 has the shortest exposure time, the imaging device can generate an image of the brightest part of the image object using the shortest exposure time t1 within the first exposure environment t1, allowing the imaging device to... Figure 11 The image data for the brightest part is generated in the third conversion gain CG3 mode.

[0120] Since the second exposure environment t2 has a longer exposure time t2 than the shortest exposure time t1, the imaging device can generate an image of the secondary bright portion of the object being imaged by using the exposure time t2. Therefore, in the second exposure environment t2, the imaging device can... Figure 11 The second conversion gain CG2 mode generates image data for the second bright or mid-tone portion.

[0121] Since the third exposure environment t3 has the longest exposure time t3, the imaging device can use the exposure time t3 to operate and generate an image of the darkest part of the object being imaged. Therefore, in the third exposure environment t3, the imaging device can... Figure 11 The image data for the darkest part is generated in the first conversion gain CG1 mode.

[0122] Image synthesis unit (see) Figure 1 (320) can use the image data of pixels running in each of the first conversion gain CG1 mode, the second conversion gain CG2 mode and the third conversion gain CG3 mode to perform the synthesis of HDR images corresponding to high dynamic range.

[0123] Therefore, in Figure 11 and Figure 12 In this process, because images in the high-resolution range are generated using conversion gains CG1, CG2, and CG3, it may be necessary to change multiple modes or take multiple photos.

[0124] Figure 13 This is a graph illustrating the generation of a high dynamic range curve using image data from each of the first, second, and third pixels of a synthetic imaging device according to one embodiment.

[0125] Figure 13 The diagram illustrates the variation of image data [LSB] for each of the first pixel PXa, the second pixel PXb, and the third pixel PXc based on light intensity (or illuminance). When different pixels at different locations within a pixel array are initially illuminated with light of spatially uniform intensity, the image data values ​​of different pixels can vary from pixel to pixel due to physical variations. For example, pixels in a pixel array can be classified into three types of pixels exhibiting three different levels of light sensitivity in response to the same input light intensity: the first pixel PXa can produce the maximum pixel output, followed by the second pixel PXb producing an intermediate pixel output, and then the third pixel PXc producing the minimum pixel output.

[0126] Based on the known light sensitivity levels and corresponding positions of pixels in the pixel array, the image synthesis unit (see...) Figure 1 (320) can use the image data of each of the first pixel PXa, the second pixel PXb and the third pixel PXc to synthesize an HDR image corresponding to high dynamic range.

[0127] In image sensing devices, each pixel has a physical limit to the amount of charge it can retain, and a pixel is said to be saturated when the amount of light illuminating it causes it to generate more charge than its charge retention limit. This is similar to the concept of a third pixel, PXc (see [link to third pixel]). Figure 7 The light intensity required to saturate Data_PXc can be greater than that required to saturate the second pixel PXb (see...). Figure 7 The required light intensity to saturate the Data_PXb, and to make the second pixel PXb (see Data_PXb) saturate the light intensity required to saturate the second pixel PXb. Figure 7 The light intensity required to saturate Data_PXb can be greater than that required to saturate the first pixel PXa (see...). Figure 7 The light intensity required for Data_PXb to saturate.

[0128] By synthesizing image data Data_PXa generated from the first pixel PXa, image signal Data_PXb generated from the second pixel PXb, and image data Data_PXc generated from the third pixel PXc, an image with high dynamic range (HDR image) can be generated. Although as Figure 11 and Figure 12As shown, various HDR imaging methods capture images at different exposure times at all pixels in a pixel array using different modes. However, the technology disclosed in this patent document captures incident light by operating the pixel array at a desired exposure time without capturing different images at different exposure times, and generates an HDR image using the measured light sensitivity levels and corresponding positions of the pixels in the pixel array. Specifically, pixel signals from pixels in the pixel array can be selected based on the pre-measured light sensitivity levels and corresponding positions of the pixels in the pixel array to compose or synthesize the final HDR image. For example, image data Data_PXa generated from the first pixel PXa can be replaced in... Figure 11 and Figure 12 In the other HDR techniques mentioned above, the image data generated in the first exposure environment t1, and the image data Data_PXb generated from the second pixel PXb can be replaced in... Figure 11 and Figure 12 In the other HDR techniques mentioned above, the image data generated in the second exposure environment t2, and the image data Data_PXc generated from the third pixel PXc can be replaced in Figure 11 and Figure 12 The image data generated in the third exposure environment t3 in the other HDR techniques mentioned above.

[0129] According to one implementation, image data Data_PXa generated from a first pixel PXa, image data Data_PXb generated from a second pixel PXb, and image data Data_PXc generated from a third pixel PXc are combined to generate a high dynamic range image (HDR image), enabling the generation of an image from a single shot with a desired exposure time without requiring multiple shots at different exposure times. In these implementations, image data Data_PXa, Data_PXb, and Data_PXc can be captured or generated simultaneously.

[0130] Figure 14 This is a flowchart illustrating a method of operating an imaging apparatus according to one embodiment. Figure 14 The operation method of the imaging device described herein involves the above references. Figures 1 to 13 The described method of operating the imaging device. Therefore, when referring to... Figure 14 When describing the operation method, you can refer to the following: Figures 1 to 13 And will omit the pair Figures 1 to 13 Repeated description of parts already described in the text.

[0131] Reference Figure 14 An operation method of an imaging apparatus according to one embodiment may include step S10, which involves capturing a pixel image of a plurality of pixels during wafer testing. In step S10, as described above... Figure 5As described, wafer testing can be performed in a white or black environment with uniform illumination across different pixel spaces to obtain the corresponding pixel output signal.

[0132] Next, the operation method of the imaging apparatus according to one embodiment includes step S20 of extracting image data of each of a plurality of pixels.

[0133] Next, the operation method of the imaging apparatus according to one embodiment includes step S30 of generating pixel identification information for each pixel by coordinates. Therefore, the pixel identification information relates to each pixel indexed by its coordinates. In this implementation, the pixel identification information indicates whether the corresponding pixel belongs to a first pixel (PXa) group with the lowest light sensitivity level corresponding to the longest exposure time t3, a second pixel (PXb) group with a medium light sensitivity level corresponding to an intermediate exposure time t2 shorter than exposure time t3, or a third pixel (PXc) group with the highest light sensitivity level corresponding to the shortest exposure time t1 shorter than exposure time t2. This pixel identification information can be generated by distinguishing multiple pixels based on image data of multiple pixels. For example, as described above... Figure 7 As described herein, when generating image data from pixels in a third pixel array sample #3 (or a pixel array according to one embodiment), the image data may include three segments having different brightness levels. For example, the image data generated from pixels PX in the third pixel array sample #3 includes first image data Data_PXa of about 50 [LSB] or greater, second image data Data_PXb between about 30 [LSB] and about 50 [LSB], and third image data Data_PXc of less than about 30 [LSB], and the pixels in the third pixel array sample #3 may be identified or classified as one of the first pixel PXa, the second pixel PXb, or the third pixel PXc corresponding to the pixel having the first image data Data_PXa, the pixel having the second image data Data_PXb, and the pixel having the third image data Data_PXc, respectively.

[0134] Next, the operation method of the imaging apparatus according to one embodiment includes step S40 of generating an HDR image. ISP (see...) Figure 1 The image synthesis unit of 300 (see 300) Figure 1 (320) can use the image data of each of the first pixel PXa, the second pixel PXb, and the third pixel PXc to synthesize an HDR image corresponding to high dynamic range. As mentioned above... Figure 13 The description has been provided, therefore the detailed description of step S40 is omitted.

[0135] Although various implementations have been described with reference to the exemplary drawings, variations and improvements can be made to the disclosed implementations and other implementations based on what is described or shown in this document.

[0136] Priority claims and references to relevant patent applications

[0137] This application claims priority and benefit to Korean Patent Application No. 10-2025-0035010, filed on March 19, 2025, the disclosure of which is incorporated herein by reference in its entirety.

Claims

1. An imaging device, the imaging device comprising: An image sensing unit includes a first pixel and a second pixel. The first pixel and the second pixel generate first image data and second image data respectively in response to receiving incident light. The size of the second image data generated from the second pixel is smaller than the size of the first image data generated from the first pixel. as well as An image signal processor (ISP) performs image processing on the first image data generated from the first pixel and the second image data generated from the second pixel. The ISP synthesizes the first image data and the second image data based on pixel information by coordinates, and the light intensity required to saturate the second pixel is greater than the light intensity required to saturate the first pixel.

2. The imaging device according to claim 1, wherein, The sensitivity of the first pixel is the same as the sensitivity of the second pixel.

3. The imaging device according to claim 2, wherein, The increase in the first image data and the second image data with exposure time is based on the sensitivity of the first pixel and the sensitivity of the second pixel, respectively.

4. The imaging device according to claim 1, wherein, At the initial illumination of the incident light, the second image data is smaller than the first image data.

5. The imaging apparatus according to claim 1, wherein the imaging apparatus further comprises: A data storage device stores the coordinate-based pixel information and sends the coordinate-based pixel information to the ISP, wherein the coordinate-based pixel information includes pixel identification information indicating whether the corresponding pixel belongs to the first pixel or the second pixel.

6. The imaging apparatus according to claim 5, wherein, The pixel identification information is determined based on pixel images obtained in a black or white environment.

7. The imaging apparatus according to claim 1, wherein, The first image data and the second image data were obtained simultaneously.

8. The imaging apparatus according to claim 7, wherein, The ISP includes an image synthesis unit that synthesizes the first image data and the second image data.

9. The imaging apparatus according to claim 1, wherein, The image sensing unit further includes a third pixel that generates third image data, the size of the third image data being smaller than the size of the second image data, and the sensitivity of the third pixel being the same as the sensitivity of the second pixel.

10. The imaging apparatus according to claim 9, wherein, At the initial illumination of the incident light, the third image data is smaller than the second image data.

11. The imaging apparatus according to claim 10, wherein, The first image data, the second image data, and the third image data are obtained simultaneously.

12. An imaging apparatus, the imaging apparatus comprising: An image sensing unit includes a first pixel that generates first image data and a second pixel that generates second image data, wherein the size of the second image data is smaller than the size of the first image data of the first pixel; as well as An image signal processor (ISP) processes the first image data generated from the first pixel and the second image data generated from the second pixel. Each of the first pixel and the second pixel includes: a photoelectric conversion element that generates photocharge in response to incident light; a transmission transistor electrically connected to the photoelectric conversion element; and a floating diffusion region connected to the transmission transistor and storing photocharge transmitted from the photoelectric conversion element through the transmission transistor.

13. The imaging apparatus according to claim 12, wherein, During the readout period, the photocharge accumulated in the photoelectric conversion element of the second pixel leaks to the transmission transistor of the second pixel.

14. The imaging apparatus according to claim 13, wherein, After the readout period, the photocharge accumulated in the floating diffusion region of the second pixel is less than the photocharge accumulated in the floating diffusion region of the first pixel.

15. A method of operating an imaging device, the method comprising the following steps: Capture pixel images of multiple pixels during wafer testing; Extract image data for each of the plurality of pixels; The multiple pixels are distinguished based on image data, and pixel identification information is generated for each pixel based on the coordinates of each pixel. as well as High dynamic range (HDR) images are generated based on the pixel recognition information.

16. The method according to claim 15, wherein, The step of generating the pixel recognition information includes: classifying the plurality of pixels into a first pixel group and a second pixel group, wherein the first pixel group includes a first pixel that generates the first image data, and the second pixel group includes a second pixel that generates the second image data, wherein the size of the second image data is smaller than the size of the first image data.

17. The method according to claim 15, wherein, The plurality of pixels have the same sensitivity, and the amount by which the image data increases with the duration of illumination is based on the sensitivity of the plurality of pixels.

18. The method of claim 15, further comprising the following steps: The pixel identification information is stored in a data storage device; as well as The pixel identification information from the data storage is provided to the image signal processor (ISP).

19. The method according to claim 18, wherein, The ISP generates the HDR image based on the pixel recognition information.

Images