Image sensor and operating method thereof

Abstract

There is provided an image sensor including: a pixel array including a plurality of pixels; a readout circuit that outputs first pixel data and second pixel data based on an output signal of the pixel array; and an image signal processor that generates third pixel data by recovering at least a portion of saturated pixel data of the first pixel data based on the second pixel data, and outputs HDR image data based on the second pixel data and the third pixel data. At least a portion of the first pixel data may have a higher intensity value than the second pixel data.

Description

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Korean Patent Application No. 10-2024-0062078, filed on May 10, 2024, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Background Technology

[0003] The exemplary embodiments of this disclosure relate to a complementary metal-oxide-semiconductor (CMOS) image sensor, and more specifically to an image sensor that generates high dynamic range (HDR) images.

[0004] An image sensor is a device that converts light signals (e.g., incident light) into electrical signals.

[0005] Image sensors are evolving toward improving image quality in various illuminance environments by enhancing dynamic range while reducing pixel size to increase resolution. Summary of the Invention

[0006] The example embodiment provides an image sensor for generating HDR images with improved signal-to-noise ratio (SNR).

[0007] According to one aspect of this disclosure, an image sensor is provided, comprising: a pixel array including a plurality of pixels; a readout circuit configured to output first pixel data and second pixel data based on an output signal of the pixel array; and an image signal processor configured to: generate third pixel data by recovering at least a portion of saturated pixel data of the first pixel data based on the second pixel data, and output high dynamic range (HDR) image data based on the second pixel data and the third pixel data, wherein a portion of the first pixel data has a higher intensity value than the second pixel data.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: an image sensor configured to output image data based on pixel signals output from a plurality of pixels; and a processor configured to receive the image data and output an image based on the image data to a display device or store the image data in a storage device, wherein the image sensor includes: a readout circuit configured to output first pixel data and second pixel data based on the pixel signals; and an image signal processor configured to: generate third pixel data by recovering at least a portion of saturated pixel data of the first pixel data based on the second pixel data, and output HDR image data based on the second pixel data and the third pixel data, wherein a portion of the first pixel data has a higher intensity value than the second pixel data.

[0009] According to another aspect of this disclosure, a method for operating an image sensor is provided, the method comprising: outputting a pixel signal and a reset signal from a pixel array comprising a plurality of pixels; outputting first pixel data and second pixel data from a readout circuit based on the pixel signal and the reset signal; generating third pixel data by recovering at least a portion of saturated pixel data of the first pixel data based on a color ratio between color channels of the first pixel data; and outputting high dynamic range (HDR) image data based on the second pixel data and the third pixel data, wherein a portion of the first pixel data has a higher intensity value than the second pixel data. Attached Figure Description

[0010] Figure 1 This is a diagram illustrating an image sensor according to one or more example embodiments.

[0011] Figure 2 This is a graph showing how the signal-to-noise ratio changes with the illumination of an HDR image.

[0012] Figure 3 This is a diagram illustrating the pixels of an image sensor according to an example embodiment.

[0013] Figure 4 yes Figure 3 The image sensor shown is a cross-sectional view.

[0014] Figure 5 This is a circuit diagram illustrating the pixels of an image sensor according to one or more example embodiments.

[0015] Figure 6A yes Figure 5 The image sensor operation timing diagram is shown.

[0016] Figure 6B yes Figure 5 The image sensor operation timing diagram is shown.

[0017] Figure 7 This is a block diagram of an image signal processor according to one or more example embodiments.

[0018] Figure 8 This is a conceptual diagram illustrating the process of recovering saturated pixels by an image signal processor according to one or more example embodiments.

[0019] Figure 9 This is a block diagram of a compensation unit according to one or more example embodiments.

[0020] Figure 10 This is a conceptual diagram illustrating the operation of demosaic blocks according to one or more example embodiments.

[0021] Figure 11This is a conceptual diagram illustrating the process of determining color ratios from image color ratio blocks according to one or more example embodiments.

[0022] Figure 12 This is a block diagram of an image signal processor according to one or more example embodiments.

[0023] Figure 13 This is a conceptual diagram illustrating the operation of a merging unit according to one or more example embodiments.

[0024] Figure 14 This is a block diagram of a compensation unit according to one or more example embodiments.

[0025] Figure 15 It is shown Figure 14 The diagram shown illustrates the concept of de-mosaicing.

[0026] Figure 16 This is a diagram illustrating the pixels of an image sensor according to one or more example embodiments.

[0027] Figure 17 This is a block diagram of an image sensor according to one or more example embodiments.

[0028] Figure 18 This is a block diagram of an image sensor according to one or more example embodiments.

[0029] Figure 19 It is a block diagram of an electronic device according to one or more example embodiments.

[0030] Figure 20 This is a flowchart illustrating a method of operating an image sensor according to one or more example embodiments. Detailed Implementation

[0031] In the following description, exemplary embodiments will be illustrated with reference to the accompanying drawings.

[0032] The following specific embodiments are provided to help readers gain a full understanding of the methods, apparatus, and / or systems described herein. However, after understanding the disclosure of this application, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become clear. For example, the order of operations described herein is merely exemplary, and this disclosure is not limited to those set forth herein, but may be changed, as will become clear upon understanding the disclosure of this application, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0033] The features described herein can be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, and many feasible ways will become clear upon understanding the disclosure of this application.

[0034] The terminology used herein is for the purpose of describing various examples only and is not intended to limit this disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprise,” “include,” and “have” indicate the presence of the stated feature, quantity, operation, component, element, and / or combination thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0035] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as would be normally understood by one of ordinary skill in the art to which this disclosure pertains. Unless expressly defined herein, terms (e.g., terms as defined in a general dictionary) shall be interpreted as having the same meaning as they have in the relevant field and in the context of this disclosure, and shall not be interpreted in an ideal or overly formalized manner.

[0036] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains, and based on an understanding of the disclosure of this application. Terms (such as those defined in common dictionaries) shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art and in the disclosure of this application, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein. The use of the term "may" in relation to examples or embodiments (e.g., what an example or embodiment may include or implement) means that there exists at least one example or embodiment that includes or implements such a feature, and that all example embodiments are not limited thereto.

[0037] The embodiments disclosed herein are exemplary embodiments, and therefore, the disclosure is not limited thereto and can be implemented in various other forms. As is conventional in the art, embodiments can be described and illustrated according to blocks as shown in the accompanying drawings, which perform one or more of the described functions. These blocks (which may be referred to herein as cells or modules, or by names such as devices, logic, circuits, counters, comparators, generators, converters, etc.) may be physically implemented by analog and / or digital circuits including one or more of logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, etc., and may also be implemented or driven by software and / or firmware (configured to perform the functions or operations described herein).

[0038] Figure 1 This is a diagram illustrating an image sensor 100 according to one or more example embodiments.

[0039] According to an embodiment, the image sensor 100 may include an image signal processor 160. The image signal processor 160 may receive at least one pixel data and output high dynamic range (HDR) image data HIMG based on the received pixel data. The image signal processor 160 may compensate for the pixel data PXD of saturated pixels and generate HDR image data HIMG based on the compensated pixel data PXD.

[0040] Now refer to Figure 1 The image sensor 100 will be described in more detail. For example, Figure 1 An example is shown of an image sensor 100 outputting pixel signals PXS in parallel for each column line. However, this disclosure is not limited thereto, and therefore, the image sensor 100 may output pixel signals in another manner. For example, the image sensor 100 may be configured to output pixel signals in parallel for each pixel.

[0041] According to an embodiment, the image sensor 100 may further include a pixel array 110, a line driver 120, a timing controller 130, a ramp signal generator 140, and a readout circuit 150. The readout circuit 150 may include an analog-to-digital converter (ADC) and an output buffer. However, this disclosure is not limited thereto, and therefore, according to another embodiment, the image sensor 100 may include one or more other components or omit one or more components.

[0042] Image sensor 100 can generate image data, which is visual information about an object captured by the lens. For example, image signal processor 160 can be configured to process pixel data received from readout circuit 150 and output the processed pixel data to a display device or store the processed pixel data in a storage device.

[0043] Pixel array 110 may include multiple pixels PX. Pixel array 110 may receive multiple pixel driving signals CS1, CS2, CS3 to CSn from row driver 120. The multiple pixel driving signals may include, but are not limited to, selection signals for controlling selection transistors, reset signals for controlling reset transistors, and transfer transistor control signals for controlling transfer transistors. Each of the multiple pixels PX in pixel array 110 can operate under the control of the received pixel driving signals CS1, CS2, CS3 to CSn.

[0044] Multiple pixels PX can be arranged, for example, in a matrix. Each pixel PX can be electrically connected to a single row line and a single column line among multiple row lines. In an example embodiment, each pixel PX may include multiple transistors controlled by row driver 120. In an example embodiment, two or more adjacent pixels PX can form a pixel group, and two or more pixels PX included in a pixel group can share at least a portion of a transmission transistor, a drive transistor, a select transistor, and a reset transistor.

[0045] Each of the plurality of pixels PX may include a photoelectric conversion element that converts incident light signals into electrical signals. Each pixel PX may include at least one photoelectric conversion element.

[0046] The photoelectric conversion element can be a photodiode (PD). Photoelectric conversion elements can include, but are not limited to, photodiodes (PDs), photocapacitors, photogates, pinned photodiodes (PPDs), partially pinned photodiodes, organic photodiodes (OPDs), quantum dots (QDs), or combinations thereof. Example embodiments are provided with reference to photodiodes (PDs) as the photoelectric conversion element; however, other photoelectric conversion elements described above can be used, and the example embodiments are not limited to photodiodes (PDs).

[0047] The row driver 120 can drive a single row or multiple rows of the pixel array 110 under the control of the timing controller 130. For example, the row driver 120 can drive at least one of the multiple rows. The row driver 120 can generate a selection signal to drive at least one of the multiple rows. The row driver 120 can activate the pixel corresponding to the selected row. The pixel signal PXS and / or reset signal RSS of the pixel in the selected row can be sent to the readout circuit 150 through multiple column lines CL1, CL2, CL3 to CLm.

[0048] The pixel signal PXS can be the voltage of the floating diffusion region. Alternatively, the pixel signal PXS can be a signal reflecting the charge generated in the photodiode PD in each of the multiple pixels. The reset signal RSS can be the voltage of the floating diffusion region used as a reference voltage to perform correlated double sampling (CDS) together with the pixel signal PXS.

[0049] The timing controller 130 can control the pixel array 110, the line driver 120, the ramp signal generator 140, and the readout circuit 150. The timing controller 130 can provide a timing control signal TC to the line driver 120.

[0050] According to the example embodiment, the timing control signal TC can be set differently based on the operating mode of the image sensor 100. For example, the image sensor 100 can operate in a normal capture mode or an HDR mode. The operating mode of the image sensor 100 can be selected by the user or set based on conditions preset by an external processor.

[0051] The line driver 120 can drive each of the multiple pixel PXs in either a normal capture mode or an HDR mode based on a timing control signal TC.

[0052] In the example scenario where the image sensor 100 operates in HDR mode based on exposure time, the row driver 120 can drive each of the plurality of pixels PX to generate at least two types of pixel signals PXS with different exposure times. For example, the row driver 120 can control each of the plurality of pixels PX to generate a second pixel signal corresponding to a second exposure time, and then control each of the plurality of pixels PX to generate a first pixel signal corresponding to a first exposure time. The first exposure time and the second exposure time can be different from each other. The first exposure time can be longer or shorter than the second exposure time.

[0053] In an example where the image sensor 100 operates in HDR mode based on conversion gain, the line driver 120 can drive each of the pixels PX based on multiple conversion gains to output a pixel signal PXS and / or a reset signal RSS. The image sensor 100 can also operate based on in-scene dual conversion gain (iDCG), which outputs a single-frame image based on multiple conversion gains. For example, the line driver 120 can control the pixels PX to generate pixel signals PXS and / or reset signals RSS by applying a high conversion gain (HCG) to the voltage of the floating diffusion region, or by applying a low conversion gain (LCG) to the voltage of the floating diffusion region. For example, a conversion gain greater than a first value can be considered a high conversion gain, and a conversion gain less than a second value can be considered a low conversion gain. The first and second values ​​can be the same. The pixels PX can output a first pixel signal and a first reset signal with a high conversion gain applied, and a second pixel signal and a second reset signal with a low conversion gain applied.

[0054] The timing controller 130 can control the ramp signal generator 140 via the ramp control signal CS_RP. The ramp control signal CS_RP may include a ramp enable signal, a mode signal, etc.

[0055] The ramp signal generator 140 can generate a ramp signal RAMP based on the ramp control signal CS_RP. For example, the ramp signal generator 140 can generate the ramp signal RAMP in response to the ramp control signal CS_RP. The ramp signal generator 140 can generate a ramp signal RAMP with a slope. For example, the ramp signal generator 140 can generate a ramp signal RAMP with a predetermined slope. The ramp signal generator 140 can provide the generated ramp signal RAMP to the ADC of the readout circuit 150.

[0056] The ADC of the readout circuit 150 can output a pixel signal PXS based on the ramp signal RAMP, and output pixel data PXD, i.e., a digital signal, based on the pixel signal PXS and the reset signal RSS. For example, the ADC can convert each of the pixel signal PXS and the reset signal RSS into a digital signal based on the ramp signal RAMP in a correlated double sampling scheme, and can output the difference between the pixel signal PXS and the reset signal RSS as the pixel data PXD, i.e., the digital signal. The pixel data PXD can be provided to the image signal processor 160. The pixel data PXD can be the intensity value corresponding to the pixel signal PXS.

[0057] According to an example embodiment, the image signal processor 160 can output HDR image data HIMG based on first pixel data and second pixel data output by the readout circuit 150.

[0058] In the example case where the image sensor 100 operates in HDR mode based on exposure time, the readout circuit 150 can output first pixel data and second pixel data based on a first pixel signal and a second pixel signal output from the pixel array 110, respectively. For example, the readout circuit 150 can output first pixel data based on the first pixel signal corresponding to the first exposure time. The readout circuit 150 can output second pixel data based on the second pixel signal corresponding to the second exposure time. The first exposure time and the second exposure time can be different from each other, and the first exposure time can be longer or shorter than the second exposure time.

[0059] In an example where the image sensor 100 operates in HDR mode based on conversion gain, the readout circuit 150 can perform correlated double sampling on a first pixel signal with applied high conversion gain (HCG) and a first reset signal, and output first pixel data. Alternatively, the readout circuit 150 can also perform correlated double sampling on a second pixel signal with applied low conversion gain (LCG) and a second reset signal, and output second pixel data. The following description provides examples of exemplary embodiments where the first pixel data is pixel data based on a first exposure time or pixel data based on high conversion gain (HCG). The following description also provides examples of exemplary embodiments where the second pixel data is pixel data based on a second exposure time or pixel data based on low conversion gain (LCG). Furthermore, the following description provides examples of exemplary embodiments where the first exposure time and the second exposure time are different from each other, and the first exposure time is longer than the second exposure time. Finally, the following description provides examples of exemplary embodiments where the high conversion gain (HCG) is a higher conversion gain than the low conversion gain (LCG).

[0060] Image signal processor 160 can receive first pixel data and second pixel data. Image signal processor 160 may include a compensation unit 161, which generates third pixel data by recovering at least a portion of the first pixel data based on the second pixel data and color ratio. Compensation unit 161 can recover at least a portion of the saturated pixel data of the first pixel data. For example, compensation unit 161 can recover all or a portion of the saturated pixel data of the first pixel data. For example, saturated pixel data may mean that the value of the pixel data is the maximum value that can be digitized by the readout circuit. Saturated pixel data may be the intensity value corresponding to the pixel signal PXS output from the saturated pixel. In this specification, the expression "recover saturated pixel" is used interchangeably with "recover the value of saturated pixel data".

[0061] In an example embodiment, the recovery of saturated pixel data may refer to the recovery of pixel data associated with a specific color channel of the saturated pixel data. In an example embodiment, the color ratio may be an estimated ratio of the unsaturated color channel to the saturated color channel of the saturated pixel data. Throughout this specification, recovery and compensation are used with the same technical meaning. Recovering pixel data can widen the dynamic range of the intensity values ​​of the pixel data.

[0062] The image signal processor 160 can generate and output HDR image data HIMG based on the second pixel data and the third pixel data. In an example embodiment, the HDR image data HIMG can be image data in which the second pixel data and the third pixel data are combined into a single data set. In an example embodiment, the HDR image data HIMG can be image data in which the second pixel data and the third pixel data exist as separate data sets. The merged HDR image data HIMG can be image data in which the second pixel data and the third pixel data are combined into a single data set, or image data in which the second pixel data and the third pixel data exist as separate data sets.

[0063] According to an example embodiment, the image sensor 100 can recover at least a portion of the saturated pixel data of the first pixel data to increase the SNR of HDR image data HIMG generated based on the first pixel data and the second pixel data.

[0064] Figure 2 This is a graph showing how the signal-to-noise ratio (SNR) changes with the illumination of an HDR image.

[0065] refer to Figure 2 The change in the SNR of an HDR image can be understood as a combination of changes in SNR1 and SNR2. The change in SNR1 is based on the high conversion gain (HCG) SNR, and the change in SNR2 is based on the low conversion gain (LCG) SNR. As illumination increases, the saturation ratio of pixel data based on the high conversion gain (HCG) may increase. For example, in an example case where a portion of the pixel signal based on the high conversion gain (HCG) begins to saturate at a first illumination level L1, more pixel signals based on the high conversion gain (HCG) may saturate at a second illumination level L2. Therefore, the saturation ratio of the pixel data may increase. Thus, an image sensor according to related technologies can generate an HDR image using at least a portion of the pixel signal based on the low conversion gain (LCG) at illumination levels of the first illumination level L1 or higher. This may result in a decrease in the SNR of the HDR image in regions with predetermined or higher illumination levels.

[0066] The image sensor 100 according to the example embodiment can recover at least a portion of the saturated pixel data of the first pixel data to recover at least a portion of the reduced SNR of the HDR image. For example, refer to Figure 2 The image sensor 100 can recover at least a portion of the saturated pixel data based on the second pixel data and color ratio, under a first illuminance L1 where a portion of the first pixel data begins to saturate based on a high conversion gain HCG. The image sensor 100 can prevent at least a portion of the SNR of the HDR image from decreasing by recovering at least a portion of the pixel data corresponding to the region from the first illuminance L1 where a portion of the first pixel data begins to saturate to a second illuminance L2 where the saturated pixel data cannot be recovered. For example, the image sensor 100 can increase the SNR of the HDR image. In the above embodiments, the first pixel data can be pixel data based on a high conversion gain HCG or a long exposure time. The second pixel data can be pixel data based on a low conversion gain LCG or a short exposure time. The range of intensity values ​​of the first pixel data can be higher than the range of intensity values ​​of the second pixel data. For example, the intensity value of at least a portion of the first pixel data can be higher than the intensity value of the second pixel data.

[0067] Figure 3 This is a diagram illustrating the pixels of an image sensor according to one or more example embodiments. According to the embodiments, Figure 3 The pixels shown can be Figure 1 The image sensor has 100 pixels per pixel (PX). For example, Figure 3 The pixels can be pixels PX corresponding to a portion of the pixel array 110 of the image sensor 100.

[0068] In an example embodiment, pixel array 110 may be a pixel array comprising pixels of different colors. For example, the pixel array may comprise repeating pixels with different color filters. Pixels with different color filters may be arranged in a Bayer pattern. Repeating pixels PX may include two pixels with a green color filter provided diagonally opposite each other, a pixel with a red color filter, and a pixel with a blue color filter. However, this disclosure is not limited thereto, and therefore, according to another embodiment, pixels may be arranged in another pattern.

[0069] Figure 4 yes Figure 3 The image sensor shown is a cross-sectional view. According to... Figure 4 The image sensor in an example embodiment may be Figure 1 Image sensor 100. Figure 4 The cross-sectional view can be along Figure 3 The pixel PX in the embodiment is cropped by the dashed line A-A'.

[0070] refer to Figure 4The image sensor 100 may include a first structure S1, a second structure S2, and a third structure S3.

[0071] In an example embodiment, the first structure S1 may include a photodiode PD, a transmission gate TG, and a first floating diffusion region FD1. For example, the first structure S1 may include multiple photodiodes PD, multiple transmission gates TG, and multiple first floating diffusion regions FD1.

[0072] In an example embodiment, the pixel circuitry for each pixel can be provided in the second structure S2. In another example embodiment, a portion of the pixel circuitry for each pixel can be provided in the first structure S1, and another portion of the pixel circuitry for each pixel can be provided in the second structure S2. For example, Figure 4 This illustrates that the first floating diffusion region FD1 of the first structure S1 is directly connected to the transistor TR of the second structure S2. However, with Figure 4 As shown, the first floating diffusion region FD1 of the first structure S1 can be electrically connected to the transistor TR of the second structure S2 through another pixel circuit of the first structure S1.

[0073] In an example embodiment, a second floating diffusion region may be provided in the second structure S2. The first floating diffusion region FD1 of the first structure S1 may be electrically coupled to or decoupled from the second floating diffusion region of the second structure S2.

[0074] In an example embodiment, the third structure S3 may include logic (or logic circuitry), such as readout circuitry, timing controller, image signal processing logic, and interface circuitry.

[0075] In an example embodiment, the first structure S1, the second structure S2, and the third structure S3 may include a wiring layer WS to transmit electrical signals.

[0076] In an example embodiment, the first structure S1 may include a first surface FS1 on a first side of the first structure S1 and a second surface BS1 on a second side of the first structure S1 opposite to the first side. The first surface FS1 may be the front surface of the first structure S1, and the second surface BS1 may be the rear surface of the first structure S1. For example, the image sensor 100 may be a back-side illumination (BSI) type image sensor, wherein light is incident on the rear surface of the first structure S1.

[0077] In an example embodiment, pixels may be provided in the first structure S1, and each pixel may include a photodiode PD, color filters CCa and CFb, and a microlens ML. Color filter CCa may be a red color filter, and color filter CFb may be a green color filter.

[0078] In an example embodiment, multiple pixel isolation layers (DTIs) can be formed between the first surface FS1 and the second surface BS1 of the first substrate W1 of the first structure S1, and can extend from the second surface BS1 toward the first surface FS1. Pixels can be separated from each other through the pixel isolation layers (DTIs).

[0079] In an example embodiment, the first structure S1 may include a device isolation portion STI.

[0080] In an example embodiment, the device isolation portion STI can extend from the first surface FS1 of the first substrate W1 toward the second surface BS1 to a predetermined depth and may include an insulating material. The device isolation portion STI may be connected to the pixel isolation layer DTI, and the boundary between the device isolation portion STI and the pixel isolation layer DTI may not be obvious.

[0081] In an example embodiment, the device isolation portion STI can be formed as a doped region extending in a direction from the first surface FS1 of the first substrate W1 toward the second surface BS1. For example, the device isolation portion STI can be formed as a doped region having a predetermined depth. The doped region can be doped with a p-type material.

[0082] In an example embodiment, the second structure S2 may include a second substrate W2.

[0083] In an example embodiment, the second substrate W2 may be a silicon-on-insulator (SOI) substrate. The SOI substrate may be bonded to the first structure S1. For example, after the SOI substrate is bonded to the first structure S1, a portion of the SOI substrate may be ground, polished, or ion-cut to be separated. The second structure S2 may include an oxide layer OX and a buried oxide layer BOX. The second substrate W2 may be referred to as the active layer.

[0084] In the example embodiment, with Figure 4 The second substrate W2 shown is different; the second substrate W2 may not include the buried oxide layer (BOX). For example, the second substrate W2 may be a general semiconductor substrate, rather than an SOI substrate.

[0085] In an example embodiment, the first structure S1 and the second structure S2 can be electrically connected to each other via a deep contact structure DCNT. The deep contact structure DCNT can be formed as a contact intersecting at least a portion of the first structure S1 and at least a portion of the second structure S2. The deep contact structure DCNT can be formed after bonding the first structure S1 and the second structure S2 to each other. In an example embodiment, the deep contact structure DCNT may include an electrical connection path formed of tungsten.

[0086] In another example embodiment, with Figure 4 Unlike the previous example, the first structure S1 and the second structure S2 can be electrically connected to each other through silicon vias.

[0087] In another example embodiment, the first structure S1 and the second structure S2 can be electrically connected to each other via Cu-Cu (C2C) bonding contacts.

[0088] In another example embodiment, the first structure S1 and the second structure S2 can be electrically connected to each other through Cu-Cu (C2C) bonding contacts, deep contact structures DCNT, and through-silicon vias.

[0089] In an example embodiment, the second structure S2 and the third structure S3 can be electrically connected via Cu-Cu (C2C) bonding contacts. In another example embodiment, with... Figure 4 Unlike the previous example, the second structure S2 and the third structure S3 can be electrically connected to each other through through-silicon vias and / or through-silicon contact (TSC).

[0090] In the example embodiment, the first surface FS1 of the first substrate W1 and the third surface BS2 of the second substrate W2 may face each other, and the fourth surface FS2 of the second substrate W2 and the fifth surface FS3 of the third substrate W3 may face each other.

[0091] Figure 5 This is a circuit diagram illustrating pixels of an image sensor according to one or more example embodiments. According to the example embodiments, Figure 5 The pixel PX can correspond to Figure 1 The image sensor has 100 pixels per 100pX. (Reference) Figure 5 The described example could be the pixel PX of an image sensor 100 operating in HDR mode based on conversion gain. However, this disclosure is not limited thereto, and therefore, according to another embodiment, the pixels of an image sensor 100 operating in HDR mode based on exposure time could have the same characteristics as... Figure 5 The example circuit configurations are different. The following embodiment will be described with reference to an example of image sensor 100 operating in HDR mode based on conversion gain.

[0092] refer to Figure 5 According to the example embodiment, a pixel PX may include a photodiode PD, a transmission transistor TG, a reset transistor RX, a drive transistor DX, a selection transistor SX, and a conversion gain transistor DRG.

[0093] The transmission transistor TG can be connected to the photodiode PD and the first floating diffusion region FD1, and can be controlled by the transmission control signal TS.

[0094] The reset transistor RX can be connected to the pixel voltage power supply VDD and the first floating diffusion region FD1 and / or the second floating diffusion region FD2, and can be controlled by the reset control signal RS. One terminal of the reset transistor RX can be connected to the pixel voltage power supply VDD, and the other terminal of the reset transistor RX can be connected to the second floating diffusion region FD2.

[0095] The switching gain transistor DRG can be connected to the first floating diffusion region FD1 and the second floating diffusion region FD2. The second floating diffusion region FD2 can be electrically coupled to the first floating diffusion region FD1 by turning on the switching gain transistor DRG. The switching gain transistor DRG can be controlled by the switching gain control signal DCS.

[0096] The driving transistor DX can be a source follower transistor and can be controlled by the voltage of the first floating diffusion region FD1. In the example case where the second floating diffusion region FD2 is electrically coupled to the first floating diffusion region FD1, the driving transistor DX can be controlled by the voltage of the first floating diffusion region FD1 coupled to the second floating diffusion region FD2. The driving transistor DX can provide an output signal amplified by the voltage supplied to the gate terminal to one terminal of the selection transistor SX.

[0097] The select transistor SX can output the signal received from the drive transistor DX to the column line CLi based on the control of the select signal SEL. The signal Vout output to the column line CLi can be a pixel signal or a reset signal.

[0098] Figure 6A and Figure 6B It is based on Figure 5 The following is an operational timing diagram of the pixel PX of the image sensor 100 in an example embodiment.

[0099] Figure 6A An example is shown where the pixel PX of the image sensor 100 outputs a pixel signal or a reset signal in the first readout mode. Figure 6B An example of the image sensor 100 outputting a pixel signal or a reset signal in the second readout mode is shown.

[0100] refer to Figure 6A and Figure 6BAccording to the example embodiment, a pixel PX can output a pixel signal or a reset signal based on a high conversion gain (HCG) and a low conversion gain (LCG). Based on the photoelectric charge of a photodiode (PD) generated by exposures over the same time period, a pixel PX can output a first pixel signal based on a high conversion gain (HCG) and a second pixel signal based on a low conversion gain (LCG). For example, based on the photoelectric charge of a photodiode (PD) generated by exposures over the same frame, a pixel PX can output a first pixel signal based on a high conversion gain (HCG) and a second pixel signal based on a low conversion gain (LCG). The charge generated by the exposure can be transferred to... Figure 5 The first floating diffusion region FD1, and can output the first pixel signal based on the charge of the first floating diffusion region FD1. Afterwards, Figure 5 The first floating diffusion region FD1 can be electrically coupled to the second floating diffusion region FD2, and a second pixel signal can be output based on the charge of the second floating diffusion region FD2 coupled to the first floating diffusion region FD1. Therefore, the pixel PX can be operated based on the scene-based dual conversion gain iDCG.

[0101] According to an example embodiment, the conversion gain transistor DRG of pixel PX can be kept in the off state to provide high conversion gain HCG and kept in the on state to provide low conversion gain LCG.

[0102] refer to Figure 6A The pixel PX that outputs a pixel signal or a reset signal in the first readout mode will now be described.

[0103] At time T1, the reset control signal RS and the conversion gain control signal DCS can go high to turn on the reset transistor RC and the conversion gain transistor DRG, and reset the floating diffusion regions FD1 and FD2. The transfer transistor TG can be in the off state. At time T2, the reset transistor RC and the conversion gain transistor DRG can be turned off.

[0104] At time T2, the selection transistor SX can be turned on by the high-level selection signal SEL. At time TA, a first reset signal based on the high conversion gain HCG corresponding to the voltage of the first floating diffusion region FD1 can be output from pixel PX.

[0105] At time T3, the transmission transistor TG can be turned on by transmitting the high-level control signal TS, and the photoelectric charge of the photodiode PD can move to the first floating diffusion region FD1. At time T4, the transmission transistor TG can be turned off. At time TB, a first pixel signal based on the high conversion gain HCG corresponding to the voltage of the first floating diffusion region FD1 can be output from pixel PX. Both the first pixel signal and the first reset signal can be output based on the high conversion gain HCG corresponding to the voltage of the first floating diffusion region FD1. Therefore, the first pixel signal and the first reset signal can be correlated with each other.

[0106] At time T5, the conversion gain transistor DRG can be turned on via a high-level conversion gain control signal DCS, and the second floating diffusion region FD2 can be electrically coupled to the first floating diffusion region FD1. The transmission transistor TG can be turned on again via a high-level transmission control signal TS. At time TC, a second pixel signal based on the low conversion gain LCG corresponding to the voltage of the first floating diffusion region FD1 electrically coupled to the second floating diffusion region FD2 can be output from pixel PX.

[0107] At time T7, the reset transistor RX can be turned on by a high-level reset control signal RS, and simultaneously the conversion gain transistor DRG is turned on. The first floating diffusion region FD1, electrically coupled to the second floating diffusion region FD2, can be reset by the pixel voltage power supply VDD. At time T8, the reset transistor RX can be turned off by a low-level reset control signal RS. At time TD, a second reset signal based on the low conversion gain LCG corresponding to the voltage of the first floating diffusion region FD1 electrically coupled to the second floating diffusion region FD2 can be output from pixel PX. Both the second pixel signal and the second reset signal can be output based on the low conversion gain LCG corresponding to the voltage of the first floating diffusion region FD1. Therefore, the second pixel signal and the second reset signal can be correlated with each other.

[0108] refer to Figure 6B The pixel PX that outputs a pixel signal or a reset signal in the second readout mode will now be described. Regarding... Figure 6A For parts that are the same as or similar to the first readout pattern described in the text, detailed descriptions will be omitted.

[0109] refer to Figure 6B , and reference Figure 6A The described operations differ; pixel PX can output a first reset signal based on a high conversion gain HCG and a second reset signal based on a low conversion gain LCG. After outputting the first and second reset signals, pixel PX can output a first pixel signal based on the high conversion gain HCG and a second pixel signal based on the low conversion gain LCG.

[0110] At time T1, the reset control signal RS and the conversion gain control signal DCS can go high to turn on the reset transistor RX and the conversion gain transistor DRG, and reset the floating diffusion regions FD1 and FD2. The transfer transistor TG is in the off state. At time T2, the reset transistor RC and the conversion gain transistor DRG can be turned off.

[0111] At time T2, the selection transistor SX can be turned on by the high-level selection signal SEL. At time TA, a first reset signal based on the high conversion gain HCG corresponding to the voltage of the first floating diffusion region FD1 can be output from pixel PX.

[0112] At time T3, the conversion gain control signal DCS goes high to turn on the conversion gain transistor DRG and electrically couple the second floating diffusion region FD2 to the first floating diffusion region FD1. At time TB, a second reset signal based on the voltage of the first floating diffusion region FD1 electrically coupled to the second floating diffusion region FD2 can be output from pixel PX, corresponding to the low conversion gain LCG.

[0113] At time T4, the conversion gain control signal DCS can switch to a low level to turn off the conversion gain transistor DRG.

[0114] At time T5, the transmission transistor TG can be turned on by transmitting the high-level control signal TS, and the photoelectric charge of the photodiode PD can move to the first floating diffusion region FD1. At time T6, the transmission transistor TG can be turned off. At time TC, a first pixel signal based on the high conversion gain HCG corresponding to the voltage of the first floating diffusion region FD1 can be output from pixel PX. Both the first pixel signal and the first reset signal can be output based on the high conversion gain HCG corresponding to the voltage of the first floating diffusion region FD1. Therefore, the first pixel signal and the first reset signal can be correlated with each other.

[0115] At time T7, the conversion gain control signal DCS can again go high to turn on the conversion gain transistor DRG and electrically couple the second floating diffusion region FD2 to the first floating diffusion region FD1. At time TD, a second pixel signal can be output from pixel PX based on the low conversion gain LCG corresponding to the voltage of the first floating diffusion region FD1 electrically coupled to the second floating diffusion region FD2. Both the second pixel signal and the second reset signal can be output based on the low conversion gain LCG corresponding to the voltage of the first floating diffusion region FD1 electrically coupled to the second floating diffusion region FD2. Therefore, the second pixel signal and the second reset signal can be correlated with each other.

[0116] Figure 7 This is a block diagram of an image signal processor according to one or more example embodiments. Figure 7 Image signal processors can correspond to Figure 1 The image signal processor 160. Now refer to... Figure 1 and Figure 7 Describe the image signal processor 160.

[0117] According to an example embodiment, the image signal processor 160 may include a compensation unit 161 and a merging unit 166. However, this disclosure is not limited thereto, and therefore, according to another embodiment, the image signal processor 160 may include one or more other components or omit one or more components.

[0118] The compensation unit 161 can receive first pixel data PD_1 and second pixel data PD_2 and output third pixel data PD_3. For example, the compensation unit 161 can output third pixel data PD_3 in which at least a portion of the saturated pixel data of the first pixel data PD_1 has been recovered. The compensation unit 161 can recover at least a portion of the saturated pixel data of the first pixel data PD_1 based on the second pixel data PD_2 and the color ratio. The first pixel data can be pixel data based on high conversion gain HCG or long exposure time. The second pixel data can be pixel data based on low conversion gain LCG or short exposure time.

[0119] The first pixel data PD_1 can be obtained from the already received reference. Figure 6B The described signal is the output of the readout circuit 150 based on the first pixel signal and the first reset signal of the high conversion gain HCG. The second pixel data PD_2 can be obtained from the received reference signal. Figure 6B The signal output by the readout circuit 150 based on the second pixel signal and the second reset signal of the low conversion gain LCG is described.

[0120] In an example embodiment, noise removal processing can be performed on the second pixel data PD_2 before it is input to the compensation unit 161. The noise-removed second pixel data PD_2 can then be provided to the compensation unit 161 and the merging unit 166.

[0121] In an example embodiment, the second pixel data PD_2 can be normalized before being input to the compensation unit 161. For example, the second pixel data PD_2 can be normalized based on the first pixel data PD_1. For example, the intensity value of the second pixel data PD_2 can be amplified by the ratio of high conversion gain HCG to low conversion gain LCG (HCG / LCG). In another example embodiment, the second pixel data PD_2 can be normalized based on the dynamic range of the intensity value of the first pixel data PD_1.

[0122] Pixel data PD_1 and PD_2 can be digital signals based on pixel signals, which are analog values ​​output from pixel PX. Pixel data PD_1 and PD_2 can also have intensity values; both PD_1 and PD_2 are digital signals.

[0123] The merging unit 166 can output HDR image data HIMG based on the second pixel data PD_2 and the third pixel data PD_3, in which at least a portion of the saturated pixel data has been recovered. The second pixel data PD_2 can be normalized pixel data.

[0124] In an example embodiment, the merging unit 166 can output HDR image data HIMG, wherein the second pixel data and the third pixel data are combined into a single data. For example, the merging unit 166 can generate HDR image data HIMG based on a value obtained by multiplying the intensity value of the second pixel data PD_2 by a first weight and a value obtained by multiplying the intensity value of the third pixel data PD_3 by a second weight. The first weight and the second weight can be predetermined values.

[0125] In the example case where the first weight is set to a value "a" between 0 and 1, the second weight can be set to "1-a". In another example embodiment, the first weight can be 0 or 1, and the second weight can be 1 or 0. In the example case where the intensity value of the third pixel data PD_3 is an unsaturated value or a value recovered from saturation, the first weight can be 0 and the second weight can be 1. In the example case where the intensity value of the third pixel data PD_3 is saturated and not recovered, the first weight can be 1 and the second weight can be 0.

[0126] In an example embodiment, the merging unit 166 can output HDR image data HIMG, where the second pixel data and the third pixel data exist as separate data. For example, the merging unit 166 can also output the second pixel data and the third pixel data alternately or in parallel.

[0127] although Figure 7 An example embodiment is shown in which the merging unit 166 outputs HDR image data HIMG by combining second pixel data and third pixel data. However, this disclosure is not limited thereto, and therefore, according to another embodiment, the merging unit 166 may output HDR image data HIMG based on second pixel data PD_2 and third pixel data PD_3 in which at least a portion of saturated pixel data has been recovered, using various merging methods.

[0128] In an example where each of the second and third pixel data is de-mosaiced pixel data, the merging unit 166 can de-mosaic the second and third pixel data, and then merge the de-mosaiced first and second pixel data. In another example embodiment, the merging unit 166 can merge the second and third pixel data, and then de-mosaic the merged first and second pixel data. HDR image data (HIMG) can be mosaicked image data.

[0129] Figure 8 This is a conceptual diagram illustrating the process of recovering saturated pixels by an image signal processor according to one or more example embodiments. (Reference) Figure 8 The recovery of the described saturated pixels can be achieved by Figures 1 to 7 The image signal processor 160 executes. Figure 7 The compensation unit 161 can perform saturation pixel restoration. Now, refer to... Figure 7 and Figure 8 The restoration of saturated pixels performed by compensation unit 161 is conceptually described.

[0130] refer to Figure 8 The compensation unit 161 can recover the intensity value of the saturated pixel SPX of the first pixel data PD_1 based on at least one of the first color ratio CR1 and the second color ratio CR2. This is described with reference to the example where the saturated pixel SPX is a pixel including a blue filter. Figure 8 Examples of embodiments are provided. However, this disclosure is not limited thereto, and therefore, the recovered pixel and the intensity value of the pixel data recovered based on the pixel signal output from the pixel can be used interchangeably. For example, a saturated pixel SPX can be a pixel including a red filter or a green filter.

[0131] In an example embodiment, the first pixel data PD_1 can be input to the compensation unit 161, wherein each unit can be stored in a row memory. For example, Figure 8 The example shows 7×7 first pixel data PD_1 stored in seven rows of memory, but the example embodiment is not limited thereto.

[0132] The compensation unit 161 can demosaic the first pixel data PD_1 to generate sub-pixels RSPX, GSPX, and SSPX corresponding to the saturated pixel SPX. Sub-pixels RSPX and GSPX can correspond to the red channel and green channel, respectively. Sub-pixels RSPX and GSPX can be generated by demosaicing based on the intensity values ​​of neighboring pixels of the saturated pixel SPX. The intensity value of the blue sub-pixel SSPX can be the same as the intensity value of the saturated pixel SPX. For example, the intensity value of the sub-pixel SSPX corresponding to the color filter of the saturated pixel SPX can be the same as the intensity value of the saturated pixel SPX.

[0133] Color channels can correspond to color filters. For example, they can exist and include color filters. Figure 1 The color channel corresponding to the color filter in pixel PX of pixel array 110. Pixel PX includes, according to... Figure 3 In the example case of the Bayer mode's red, green, and blue color filters, red subpixels of the red channel, green subpixels of the green channel, and blue subpixels of the blue channel can be generated by demosaicing.

[0134] The example embodiment is based on the finding that, depending on the light source and the object, the color channels of an image sensor are saturated differently for each color channel. For example, even when a single color channel is saturated, information about the saturated color channel can be preserved in the other color channels. Therefore, according to the example embodiment, saturated color channels can be recovered based on the color ratios that represent the relationships between color channels.

[0135] refer to Figure 8 In an example embodiment, the color ratio can be the ratio of the intensity values ​​of the sub-pixels RSPX, GSPX, and SSPX corresponding to the saturated pixel SPX of color channels CH_1, CH_2, and CH_3. For example, the first color ratio CR_1 can be the ratio of the intensity value of the unsaturated blue sub-pixel to the intensity value of the green sub-pixel GSPX. The second color ratio CR_2 can be the ratio of the intensity value of the unsaturated blue sub-pixel to the intensity value of the red sub-pixel RSPX.

[0136] In the example embodiment, the intensity value of the blue sub-pixel SSPX, which is the sub-pixel corresponding to the color filter of the saturated pixel SPX, can be recovered based on the following Equation 1.

[0137] Equation 1

[0138] The estimated intensity value of SSPX = First coefficient × CR_1 × intensity value of GSPX + Second coefficient × CR_2 × intensity value of RSPX

[0139] In an example embodiment, the estimated value can be used as the intensity value of the unsaturated blue sub-pixel SSPX to calculate the first color ratio CR_1 and the second color ratio CR_2. For example, the intensity value of the blue sub-pixel of the second pixel data corresponding to the saturated pixel SPX of the first pixel data PD_1 can be estimated as the intensity value of the unsaturated blue sub-pixel.

[0140] In an example embodiment, the estimated values ​​can be used as the intensity values ​​of the unsaturated blue sub-pixels, green sub-pixels, and red sub-pixels to calculate the first color ratio CR_1 and the second color ratio CR_2. For example, the intensity values ​​of the blue, green, and red sub-pixels of the second pixel data corresponding to the saturated pixel SPX of the first pixel data PD_1 can be used to calculate the first color ratio CR_1 and the second color ratio CR_2.

[0141] In the example embodiment, the intensity values ​​of the sub-pixels used to calculate the first color ratio CR_1 and the second color ratio CR_2 can be values ​​that have already been filtered by a low-pass filter. For example, the intensity values ​​can be values ​​that have already undergone blurring. For instance, the intensity values ​​of each of the green sub-pixel GSPX of the first pixel data and the blue sub-pixel of the second pixel data can be used to calculate the first color ratio CR_1 after blurring. Similarly, the intensity values ​​of each of the red sub-pixel RSPX of the first pixel data and the blue sub-pixel of the second pixel data can be used to calculate the second color ratio CR_2 after blurring. In the example case where values ​​that have already been filtered by a low-pass filter are used to calculate the color ratio, the intensity values ​​of the sub-pixels can be used to prevent inaccuracies in the color ratio values ​​caused by abnormal local variations.

[0142] In the example embodiment, the first and second coefficients of Equation 1 can be values ​​between 0 and 1. For example, each of the first and second coefficients can have a value of 0.5.

[0143] In the example where pixels adjacent to the saturated pixel SPX of the first pixel data PD_1 are also saturated, the color channel corresponding to the saturated adjacent pixels may not be used to recover the saturated pixel SPX. In the example where green pixels adjacent to the saturated pixel SPX of the first pixel data PD_1 are saturated, the first coefficient may have a value of 0. Therefore, in the example where all pixels adjacent to the saturated pixel SPX of the first pixel data PD_1 are saturated, the saturated pixel SPX may not be recovered. For example, if both green and red pixels adjacent to the saturated pixel SPX of the first pixel data PD_1 are saturated, the saturated pixel SPX may not be recovered.

[0144] An example has already been described regarding the saturated pixel SPX of the first pixel data PD_1, which includes the blue filter. Figure 8 Example implementation. However, in the example case where the saturated pixel is a pixel that includes a color filter of another color, the saturated pixel SPX can be recovered based on the following Equation 2. In Equation 2, the first sub-pixel is the sub-pixel of the first color channel associated with the saturated pixel.

[0145] Equation 2

[0146] The estimated intensity value of the first sub-pixel of a saturated pixel = first coefficient × CR_A × intensity value of the second sub-pixel + second coefficient × CR_B × intensity value of the third sub-pixel

[0147] In Equation 2, CR_A (color ratio A) is the ratio of the first color channel to the second color channel, and CR_B (color ratio B) is the ratio of the first color channel to the third color channel. The second and third color channels are color channels associated with the color filter array that are not associated with saturated pixels. In an example embodiment, the color ratio between color channels can be calculated as the intensity ratio between sub-pixels of the color channel. For example, color ratio A (CR_A) can be the ratio of the intensity value of a first sub-pixel of the first color channel to the intensity value of a second sub-pixel of the second color channel. Color ratio B (CR_B) can be the ratio of the intensity value of a first sub-pixel of the first color channel to the intensity value of a third sub-pixel of the third color channel. As described above, at least a portion of the intensity values ​​of the sub-pixels used to calculate color ratio A (CR_A) and color ratio B (CR_B) can be estimated using the intensity values ​​of the sub-pixels of the second pixel data. For example, to calculate color ratio A (CR_A) and color ratio B (CR_B), the intensity value of the first sub-pixel of the first color channel can be estimated as the intensity value of the first sub-pixel of the first color channel of the second pixel data.

[0148] In an example embodiment, the intensity value of the sub-pixel used to calculate the color ratio can be the value after a low-pass filter has been applied. For example, to calculate color ratio A (CR_A) and color ratio B (CR_B), the intensity value of the second sub-pixel of the second color channel and the intensity value of the third sub-pixel of the third color channel can be the intensity value after a low-pass filter has been applied to each of the second and third sub-pixels.

[0149] Figure 9 This is a block diagram of a compensation unit according to one or more example embodiments. Figure 9 The compensation unit can correspond to Figure 7 Compensation unit 161. Now refer to Figure 9 , Figure 10 and Figure 11 Description of compensation unit 161.

[0150] The compensation unit 161 may include a de-mosaic block 162, a color ratio block 163, and a saturation compensation block 164.

[0151] De-mosaic block 162 can receive the first pixel data PD_1 and the second pixel data PD_2.

[0152] The first pixel data PD_1 can be pixel data based on high conversion gain HCG or long exposure time. The second pixel data PD_2 can be pixel data based on low conversion gain LCG or short exposure time. Each of the first pixel data PD_1 and the second pixel data PD_2 can be pixel data with a high conversion gain HCG or long exposure time. Figure 1 The pixel data of the mosaic pattern of the color filter array of the pixel array 110.

[0153] In an example embodiment, the demosaic block 162 may include a normalization block 162_1 that normalizes the second pixel data PD_2. The second pixel data PD_2 may be normalized based on the first pixel data PD_1. For example, each intensity value of the second pixel data PD_2 may be amplified by the ratio of high conversion gain HCG to low conversion gain LCG (HCG / LCG). In another example embodiment, the second pixel data PD_2 may be normalized based on the dynamic range of the intensity values ​​of the first pixel data PD_1.

[0154] In an example embodiment, demosaic block 162 can perform demosaicing and low-pass filtering on first pixel data PD_1 and second pixel data PD_2. Demosaic block 162 can perform demosaicing on each color channel. For example, each demosaic convolution filter and blur convolution filter associated with a color channel can be convolved with the pixel data PD of each color filter. For example, refer to Figure 10 The demosaic convolutional filter (DMSC) and the blur convolutional filter (BLUR) for each color channel can be convolved with pixel data (PD) stored in multiple row memories. The demosaic convolutional filter (DMSC) and the blur convolutional filter (BLUR) associated with the first color channel can be convolved with pixel PX_A of the pixel data PD to generate pixel PX_B. In an example embodiment, demosaicing and low-pass filtering can be performed simultaneously on a single convolutional filter (DMSC*BLUR).

[0155] refer to Figure 10 The demosaic block 162 can convolve the first pixel data PD_1 with the demosaic convolution filter DMSC and the blur convolution filter BLUR according to each color channel, such as... Figure 11As shown, pixel data DPD_1R as the red channel of the first color channel, pixel data DPD_1G as the green channel of the second color channel, and pixel data DPD_1B as the blue channel of the third color channel are generated. In the example embodiment, the coefficients of the demosaic convolution filter DMSC and the blur convolution filter BLUR can be different for each color channel. An example is provided of associating the saturated pixel SPX of the first pixel data PD_1 with the blue color channel. However, this disclosure is not limited thereto, and therefore, according to another embodiment, the saturated pixel SPX of the first pixel data PD_1 can be associated with either the red channel or the green channel. The demosaic block 162 can also convolve the demosaic convolution filter DMSC and the blur convolution filter BLUR with the second pixel data PD_2 according to each color channel, as shown. Figure 11 As shown, pixel data DPD_2R is generated as the red channel (first color channel), pixel data DPD_2G as the green channel (second color channel), and pixel data DPD_2B as the blue channel (third color channel). The intensity value range of the first pixel data PD_1 can be higher than the intensity value range of the second pixel data PD_2. For example, the first pixel data PD_1 can be pixel data based on high conversion gain HCG or long exposure time. The second pixel data PD_2 can be pixel data based on low conversion gain LCG or short exposure time.

[0156] refer to Figure 11 Color block 163 can receive de-mosaic first pixel data DPD_1 and normalized de-mosaic second pixel data DPD_2. The de-mosaic first pixel data DPD_1 may include pixel data DPD_1R, DPD_1G, and DPD_1B for each color channel. The normalized de-mosaic second pixel data DPD_2 may include pixel data DPD_2R, DPD_2G, and DPD_2B for each color channel.

[0157] Color ratio block 163 can determine at least one color ratio based on the de-mosaic first pixel data DPD_1 and the de-mosaic second pixel data DPD_2, each of which has been de-mosaiced.

[0158] The color ratio CR can include at least one color ratio between multiple color channels corresponding to a color filter array. For example, a saturated pixel of the first pixel data PD_1 can be associated with a third color channel, and the color filter array can include a color filter for the first color channel, a color filter for the second color channel, and a color filter for the third color channel. The color ratio can include at least one of a first color ratio and a second color ratio, wherein the first color ratio is the color ratio of the third color channel to the first color channel, and the second color ratio is the color ratio of the third color channel to the second color channel.

[0159] For example, used for recovery Figure 11 The color ratio of the saturated pixel SPX of the first pixel data PD_1 may include at least one of a first color ratio and a second color ratio.

[0160] The first color ratio can be the intensity value of the first sub-pixel RSPX of the red channel DPD_1R of the first color channel of the de-mosaiced first pixel data DPD_1, and the intensity value of the third sub-pixel BLPX of the blue channel DPD_2B of the third color channel of the normalized de-mosaiced second pixel data DPD_2. The intensity values ​​of the first sub-pixel RSPX and the third sub-pixel BLPX can be the intensity values ​​after performing low-pass filtering on the first sub-pixel RSPX and the third sub-pixel BLPX, respectively.

[0161] The second color ratio can be the intensity value of the second sub-pixel GSPX of the green color channel DPD_1G of the second color channel of the de-mosaiced first pixel data DPD_1, and the intensity value of the third sub-pixel BLPX of the blue color channel DPD_2B of the third color channel of the normalized de-mosaiced second pixel data DPD_2. The intensity values ​​of the second sub-pixel GSPX and the third sub-pixel BLPX can be the intensity values ​​after performing low-pass filtering on the second sub-pixel GSPX and the third sub-pixel BLPX, respectively.

[0162] The color ratio CR can be different for each saturated pixel of the first pixel data PD_1. In the example case where the first and second pixels of the first pixel data PD_1 are saturated and are associated with the same first color channel, the first color ratio of the first pixel and the first color ratio of the second pixel can be different from each other. (Reference) Figure 11 The first sub-pixel RSPX and the second sub-pixel GSPX can be used. The first sub-pixel RSPX and the second sub-pixel GSPX are pixels corresponding to the saturated pixel SPX used to recover the saturated pixel SPX. For example, the sub-pixel corresponding to the saturated pixel of the first pixel data PD_1 can be used to calculate the color ratio. Therefore, the color ratios used to recover the saturated pixel can be different from each other.

[0163] The saturation compensation block 164 can generate a third pixel PD_3 based on the color ratio CR and the de-mosaiced first pixel data DPD_1, wherein at least a portion of the saturated pixels of the first pixel data PD_1 has been recovered. For example, the saturation compensation block 164 can recover saturated pixels using Equation 1 and / or Equation 2, as referenced. Figure 8 As stated above.

[0164] Saturation compensation block 164 can recover the saturated pixel SPX of the first pixel data PD_1 based on the color ratio. For example, refer to Figure 11 The saturation compensation block 164 can recover the blue sub-pixel SSPX based on the color ratio. The blue sub-pixel SSPX is a sub-pixel of the color channel associated with the blue pixel that is the saturated pixel SPX. The saturation compensation block 164 can use a reference... Figure 11 The first color ratio and the second color ratio are described. The saturation compensation block 164 can recover the blue sub-pixel SSPX by applying the first color ratio, the second color ratio, the intensity value of the first sub-pixel RSPX (which is the red color channel DPD_1R as the first color channel), and the intensity value of the second sub-pixel GSPX (which is the green color channel DPD_1G as the second color channel) to Equation 2, as shown in the reference. Figure 11 As stated above.

[0165] Figure 12 This is a block diagram of an image signal processor 160a according to one or more example embodiments.

[0166] Figure 12 Image signal processors can correspond to Figure 1 The image signal processor 160. Now refer to... Figure 12 and Figure 13 Description of Image Signal Processor 160. Regarding... Figures 7 to 11 The detailed descriptions of similar or redundant parts will be omitted.

[0167] The image signal processor 160a according to the example embodiment may include a first merging unit 167, a compensation unit 161a, and a second merging unit 168.

[0168] The first merging unit 167 can receive first pixel data PD_1 and second pixel data PD_2, and can provide fourth pixel data PD_4 to the compensation unit 161a. For example, the first merging unit 167 can output fourth pixel data PD_4 by merging the first pixel data PD_1 and the second pixel data PD_2 into a single data.

[0169] In the example embodiment, noise removal processing can be performed on the fourth pixel data PD_4. For example, compared with the reference... Figure 7The described embodiment differs in that noise removal processing can be performed after the first pixel data PD_1 and the second pixel data PD_2 are merged. Therefore, the row memory required to separately store the first pixel data PD_1 and the second pixel data PD_2 can be reduced. Furthermore, it is not necessary to synchronize the row delay of the first pixel data PD_1 with that used for noise removal processing of the second pixel data PD_2, thereby simplifying the circuitry. In addition to noise removal processing, processing to improve image quality can also be performed on the fourth pixel data PD_4, where the first pixel data PD_1 and the second pixel data PD_2 are merged.

[0170] The compensation unit 161a can receive fourth pixel data PD_4, in which first pixel data PD_1 and second pixel data PD_2 are merged, and can output third pixel data PD_3, in which the pixel data of saturated pixels in the fourth pixel data PD_4 is restored. The saturated pixels in the fourth pixel data PD_4 can correspond to the saturated pixels in the first pixel data PD_1. The compensation unit 161a can generate first pixel data PD_1 from the fourth pixel data PD_4 to restore saturated pixels. The compensation unit 161a can perform de-mosaicing and low-pass filtering on the fourth pixel data PD_4, and can also perform de-mosaicing on the first pixel data PD_1. (See below for reference.) Figure 14 and Figure 15 The compensation unit 161a is described in detail.

[0171] The second merging unit 168 can merge the second pixel data in which the saturated pixels are recovered and the recovered third pixel data PD_3 to generate HDR image data HIMG. The second merging unit 168 can be compared with a reference... Figure 7 The merging unit 166 described herein operates in the same manner.

[0172] Figure 13 It shows the basis Figure 12 A conceptual diagram of the operation of the first merging unit 167 in an example embodiment.

[0173] refer to Figure 13 The second pixel data PD_2 can be normalized by normalization block 167_1 before merging is performed. In the example case where the first pixel data PD_1 is pixel data based on high conversion gain HCG and the second pixel data PD_2 is pixel data based on low conversion gain LCG, each intensity value of the second pixel data PD_2 can be amplified by the ratio of high conversion gain HCG to low conversion gain LCG (HCG / LCG).

[0174] In the example embodiment, reference Figure 13The first merging unit 167 can replace the pixel data of the saturated pixel SPX of the first pixel data PD_1 with the pixel data of the normalized second pixel data PD_2. In the example case where the pixel data is 10-bit data, the first merging unit 167 can replace the intensity value of the first pixel data PD_1, which has an intensity value of 1024, in the saturated pixel SPX with the pixel data of the second pixel data PD_2. The first merging unit 167 can replace the intensity value of the saturated pixel SPX with the pixel data of the pixel SLPX of the second pixel data PD_2 corresponding to the saturated pixel SPX. Therefore, the fourth pixel data PD_4 can be the pixel data in which the intensity value of the saturated pixel SPX of the first pixel data PD_1 is replaced with the intensity value of the corresponding pixel SLPX of the normalized second pixel data PD_2.

[0175] Now refer to Figure 14 Description based on Figure 12 The compensation unit 161a in the example embodiment.

[0176] The compensation unit 161a may include a de-mosaic block 162a, a color ratio block 163, and a saturation compensation block 164.

[0177] Remove mosaic blocks 162a from Figure 12 The first merging unit 167 receives the fourth pixel data PD_4. The fourth pixel data PD_4 can be the pixel data in which the first pixel data PD_1 and the second pixel data PD_2 are merged, such as... Figure 13 As shown.

[0178] The demosaic block 162a may include a first pixel data generator 162a_1. The first pixel data generator 162a_1 may generate first pixel data PD_1 based on fourth pixel data PD_4. In the example case where the pixel data is 10-bit data, the first pixel data generator 162a_1 may generate first pixel data PD_1 from fourth pixel data PD_4 as multiple pixel data with an intensity value less than 1024.

[0179] Demosaic block 162a can perform demosaicing and / or low-pass filtering (e.g., perform a low-pass filtering operation) on the generated first pixel data PD_1 and fourth pixel data PD_4 respectively. For example, demosaic block 162a can demosaic the first pixel data PD_1 to generate demosaiced first pixel data DPD_1, and can perform demosaicing and low-pass filtering on the fourth pixel data PD_4 to generate demosaiced fourth pixel data DPD_4. Demosaic block 162a can provide the demosaiced first pixel data DPD_1 to saturation compensation block 164, and provide the demosaiced fourth pixel data DPD_4 to color ratio block 163.

[0180] Color ratio block 163 can calculate at least one color ratio CR, as shown in the reference. Figure 11 As described, at least one calculated color ratio CR can be provided to the saturation compensation block 164.

[0181] Saturation compensation block 164 can generate third pixel data PD_3, where the saturated pixels of the de-mosaiced first pixel data DPD_1 are restored, based on at least one color ratio CR and de-mosaiced first pixel data DPD_1, as shown in the reference. Figure 11 As stated above.

[0182] Figure 15 It shows the basis Figure 14 A conceptual diagram of the operation of demosaic block 162a in an example embodiment.

[0183] In an example embodiment, demosaic block 162a can perform demosaicing and low-pass filtering on the fourth pixel data PD_4. For example, similar to Figure 11 The illustration shows that the demosaic convolution filter and low-pass convolution filter associated with each color channel can be convolved with the fourth pixel data PD_4. In an example embodiment, the low-pass convolution filter can be a blurring convolution filter.

[0184] Demosaic block 162a can perform demosaicing on the first pixel data PD_1. For example, similar to Figure 11 The illustration shows that the demosaic convolution filter associated with each color channel can be convolved with the first pixel data PD_1.

[0185] The fourth pixel data PD_4 may include the pixel data of the saturated pixel SPX of the first pixel data, which is replaced by the normalized value of the second pixel data PD_2, such as... Figure 13 As described above, the demosaiced fourth pixel data DPD_4 may include a first color channel DPD_4A, a second color channel DPD_4B, and a third color channel DPD_4C. An example is provided where the first color channel DPD_4A is a color channel associated with a saturated pixel SPX. Each of the color channels DPD_4A, DPD_4B, and DPD_4C of the demosaiced fourth pixel data DPD_4 may include sub-pixels LPXA, LPXB, and LPXC corresponding to the saturated pixel SPX.

[0186] The first pixel data DPD_1 for demosaicing may include a first color channel DPD_1A, a second color channel DPD_1B, and a third color channel DPD_1C. An example is provided where the first color channel DPD_1A is a color channel associated with a saturated pixel SPX. Each of the color channels DPD_1A, DPD_1B, and DPD_1C of the first pixel data DPD_1 for demosaicing may include sub-pixels SPXA, SPXB, and SPXC corresponding to the saturated pixel SPX.

[0187] The color channels DPD_4A, DPD_4B, and DPD_4C of the fourth pixel data DPD_4 (de-mosaiced) can be provided to color ratio block 163. The color channels DPD_1A, DPD_1B, and DPD_1C of the first pixel data DPD_1 (de-mosaiced) can be provided to saturation compensation block 164.

[0188] Color ratio block 163 can determine at least one color ratio CR based on the color channels DPD_4A, DPD_4B and DPD_4C of the fourth pixel data DPD_4.

[0189] In the example, the first color channel DPD_4A is the color channel associated with the saturated pixel, such that the pixel LPXA of the first color channel DPD_4A of the fourth pixel data DPD_4 can be a pixel in which the saturated pixel of the first pixel data PD_1 is replaced with the normalized value of the second pixel data PD_2. The second color channel DPD_4B and the third color channel DPD_4C of the fourth pixel data DPD_4 can be unsaturated color channels, such that the pixels LPXB and LPXC corresponding to the saturated pixel SPX can be pixels that have been de-mosaiced from the unsaturated pixels of the first pixel data PD_1.

[0190] Therefore, the first color ratio, which is the ratio of the first color channel to the second color channel, can be calculated as the ratio of the intensity value of pixel LPXA to the intensity value of pixel LPXB. Furthermore, the second color ratio, which is the ratio of the first color channel to the third color channel, can be calculated as the ratio of the intensity value of pixel LPXA to the intensity value of pixel LPXC.

[0191] Figure 16 This is a diagram illustrating the pixels of an image sensor according to one or more example embodiments. Reference Figure 16 The described pixels can correspond to Figure 1 The image sensor has 100 pixels (PX). Now, referencing... Figure 1 and Figure 16 Describes pixel PX.

[0192] In an example embodiment, pixel array 110 may be a pixel array in which pixels PX having the same color filter are arranged consecutively. Pixels including the same color filter may be referred to as a pixel group. Figure 16 The pixels PX can be arranged in a tetra-cell structure. An example is described where four pixels with the same color filter form a single pixel group. The first pixel group PG1 and the fourth pixel group PG4 may include a green color filter, the second pixel group PG2 may include a red color filter, and the third pixel group PG3 may include a blue color filter. For example, pixel groups PG1, PG2, PG3, and PG4 can be arranged in a Bayer pattern. Figure 16 The same pixel groups PG1, PG2, PG3 and PG4 shown can be repeated in pixel array 110.

[0193] Including according to Figure 16 The image sensor 100 of the pixel array 110 in the example embodiment can operate in tetra mode or normal mode.

[0194] In normal mode, pixel array 110 can output each of the pixels included in pixel groups PG1, PG2, PG3 and PG4 as a pixel signal of analog signal.

[0195] In an example embodiment, in quad-mode, pixel array 110 can output as an analog signal for each of pixel groups PG1, PG2, PG3, and PG4, instead of outputting an analog signal from each pixel. For example, pixel array 110 can operate on a pixel group basis. For example, pixels included in the same pixel group can be controlled in the same manner, and signals output from pixels included in the same pixel group can be binned. In quad-pixel mode, readout circuit 150 can output first pixel data and second pixel data. Image signal processor 160 can recover pixel data of the saturated pixel group from the first pixel data of pixel groups PG1, PG2, PG3, and PG4 operating in quad-pixel mode. Image signal processor 160 can recover at least a portion of the pixel data of the saturated pixel group of the first pixel data based on the second pixel data and at least one color ratio.

[0196] In an example embodiment, a pixel group may include pixels of the form M*N (where M and N are integers greater than or equal to 2). The M*N form may be a configuration in which M pixel lines are arranged vertically. Each of the M pixel lines may include N pixels arranged consecutively in the horizontal direction. (See reference...) Figure 12In the described example embodiment, M and N are 2, but the example embodiment is not limited to this, and M and N can be 3 or greater. Furthermore, M and N can be different integers from each other.

[0197] Figure 17 This is a block diagram of an image sensor 100a according to one or more example embodiments. Detailed descriptions of redundant parts will be omitted.

[0198] Image sensor 100a may include a stacked first substrate 10a and a second substrate 20a. The first substrate 10a and the second substrate 20a may be connected to each other via a wafer bonding process using pixel-level C2C interconnects. The first substrate 10a and the second substrate 20a may be electrically connected not only via in-pixel contacts IN_CT within the pixel PXa, but also via a Cu-to-Cu (C2C) array provided in the peripheral region of the substrates. Control signals for controlling pixel circuitry may be transmitted via the C2C array. Pixel signals or pixel data from the first substrate 10a may be transmitted via the in-pixel contacts IN_CT to the readout circuitry or image signal processor of the second substrate 20a.

[0199] Figure 18 This is a block diagram of an image sensor 100b according to one or more example embodiments. Detailed descriptions of redundant parts will be omitted.

[0200] refer to Figure 18 The image sensor 100b may include a first substrate 10b, a second substrate 20b, and a third substrate 30b. The third substrate 30b, the second substrate 20b, and the first substrate 10b may be stacked sequentially in a direction D3 perpendicular to the plane of the substrates (parallel to the surfaces of D1 and D2).

[0201] In an example embodiment, pixel circuits PXb_1, PXb_2, and PXb_3 can be formed on a first substrate 10b and a second substrate 20b, respectively. Among pixel circuits PXb_1, PXb_2, and PXb_3, a first portion of circuit PXb_1 can be disposed on the first substrate 10b, and the second portions of circuit (the remaining circuits) PXb_2 and PXb_3 can be disposed on the second substrate 20b. The third substrate 30b may include logic (such as readout circuitry, a timing controller, or an image signal processor) and interface circuitry. The readout circuitry may include an analog-to-digital converter (ADC).

[0202] For example, a photodiode and a transmission transistor can be provided on a first substrate 10b, and the remaining pixel circuitry can be provided on a second substrate 20d.

[0203] The circuitry constituting the pixel is provided on the first substrate 10b and the second substrate 20b in a form that is not limited to this.

[0204] The first substrate 10b and the second substrate 20b can be electrically connected to each other.

[0205] In an example embodiment, the first substrate 10b and the second substrate 20b can transmit pixel signals or control signals through through-silicon vias (TSVs) provided in the peripheral regions of the substrates.

[0206] In the example embodiment, the first portion of the circuit PXa_1 on the first substrate 10b and the second portion of the circuit PXb_2 on the second substrate 20b can also be electrically connected through the first substrate interconnect structure INTC_1. The substrate interconnect structure INTC_1 can be a Cu-to-Cu (C2C) bonding contact or a deep contact structure. The deep contact structure can include a through-silicon via (TSV). The substrate interconnect structure INTC_1 can electrically connect the intra-pixel contact IN_CT1 and the intra-pixel contact IN_CT2. The intra-pixel contact IN_CT1 is electrically connected to the elements of the first portion of the circuit PXa_1, and the intra-pixel contact IN_CT2 is electrically connected to the elements of the second portion of the circuit PXb_2.

[0207] In an example embodiment, the first substrate 10b and / or the second substrate 20b can be electrically connected to the third substrate 30b via a through-silicon via (TSV) and / or an inter-substrate interconnect structure (INTC_2). Signals from the first substrate 10b and / or the second substrate 20b can be transmitted to the readout circuitry (or image signal processor) of the third substrate 30b via the TSV and / or the inter-substrate interconnect structure (INTC_2).

[0208] In the example embodiment, the second part of the circuit PXb_2 can be electrically connected to the circuit of the third substrate 30b via Cu-to-Cu (C2C) bonding contacts. The second substrate connection structure INTC_2 may include Cu-to-Cu (C2C) bonding contacts.

[0209] In the example embodiment, the third part of the circuit PXb_3 can be electrically connected to the circuit of the third substrate 30b via through-silicon copper (TSC).

[0210] Figure 19 This is a block diagram of an electronic device according to one or more example embodiments. Detailed descriptions of redundant parts will be omitted.

[0211] The electronic device 1000 may include a capture unit 1100, an image sensor 1200, a processor 1300, a display device 1400, and a storage device 1500.

[0212] The processor 1300 can control the overall operation of the electronic device 1000. The processor 1300 can provide control signals to the lens driver (actuator) 1120 to control the position of the lens 1110. Therefore, the focal length can be controlled.

[0213] The capturing unit 1100 may include a lens 1110 as a light receiving component and an actuator 1120. The lens 1110 may include multiple lenses.

[0214] The actuator 1120 can move the lens 1110 in the direction of increasing or decreasing distance from the object S based on the control signal from the processor 1300.

[0215] Image sensor 1200 can generate image data and phase data based on incident light. Image sensor 1200 may include pixel array 1210, timing controller 1220, readout circuit 1230 and image signal processor (ISP) 1240.

[0216] The pixels of pixel array 1210 may include at least one photoelectric conversion element.

[0217] The pixels of the pixel array 1210 according to the example embodiment can operate in normal mode or HDR mode. The image signal processor 1240 can generate a mode control signal MC based on the capture mode signal MODE sent by the processor 1300. The pixels can operate in normal mode or HDR mode based on the mode control signal MC sent by the image signal processor 1240.

[0218] The processor 1300 can provide a mode control signal MC to the timing controller 1220. The timing controller 1220 can control the operation of the pixel array 1210 based on the mode control signal MC.

[0219] In an example where the pixels of pixel array 1210 operate in HDR mode, processor 1300 can generate an image signal based on HDR image data HIMG provided by image sensor 1200, and display the image signal on display device 1400 or store the image signal in storage device 1500.

[0220] In an example where image sensor 1200 provides processor 1300 with first pixel data based on a long exposure and second pixel data based on a short exposure as HDR image data (HIMG), processor 1300 can combine the first pixel data and the second pixel data to generate a single image signal. At least a portion of the saturated pixels in the first pixel data can be pixel data recovered based on the second pixel data and color ratio.

[0221] In an example where the image sensor 1200 provides first pixel data and second pixel data combined into a single third pixel data as HDR image data (HIMG), the processor 1300 can display the third pixel data as an image signal on the display device 1400 or store the third pixel data in the storage device 1500. At least a portion of the third pixel data may be pixel data obtained by recovering saturated pixel data based on the first pixel data based on the high conversion gain (HCG) based on the second pixel data based on the low conversion gain (LCG) and color ratio, output by the readout circuit 1230.

[0222] Figure 20 This is a flowchart illustrating a method of operating an image sensor according to one or more example embodiments. Detailed descriptions of redundant parts will be omitted. Figure 20 The operation method of the image sensor can be determined by Figure 1 The image sensor 100 is used for execution.

[0223] In operation S110, the method may include outputting pixel signals and reset signals. For example, multiple pixels PX of the pixel array 110 of the image sensor 100 may output pixel signals and reset signals.

[0224] Each of the pixels in PX can operate in normal capture mode or HDR mode.

[0225] In an example where the image sensor 100 operates in HDR mode based on exposure time, pixel PX can output a first pixel signal corresponding to a first exposure time, and then output a second pixel signal corresponding to a second exposure time. The first exposure time and the second exposure time can be different from each other. The first exposure time can be longer or shorter than the second exposure time.

[0226] In an example where the image sensor 100 operates in HDR mode based on conversion gain, the pixel PX can output a first pixel signal and a first reset signal based on high conversion gain, and output a second pixel signal based on low conversion gain LCG.

[0227] In operation S120, the method may include outputting first pixel data and second pixel data based on a pixel signal and a reset signal. For example, the readout circuit may output first pixel data and second pixel data based on a pixel signal and a reset signal.

[0228] In operation S130, the method may include determining a color ratio between the color channels of the first pixel data, and generating third pixel data by recovering at least a portion of the saturated pixel data of the first pixel data based on the color ratio. For example, an image signal processor may determine a color ratio between the color channels of the first pixel data, and generate third pixel data by recovering at least a portion of the saturated pixel data of the first pixel data based on the color ratio. The image signal processor may output HDR image data based on the second pixel data and the third pixel data. At least a portion of the first pixel data may have a higher intensity value than the second pixel data.

[0229] As described above, according to one or more example embodiments, an image sensor can improve the signal-to-noise ratio (SNR) of HDR images.

[0230] While exemplary embodiments have been shown and described above, it will be apparent to those skilled in the art that modifications and variations may be made without departing from the scope of the inventive concept as defined by the appended claims.

Claims

1. An image sensor, comprising: A pixel array, comprising multiple pixels; The readout circuit is configured to output first pixel data and second pixel data based on the output signal of the pixel array; and The image signal processor is configured as follows: The third pixel data is generated by recovering at least a portion of the saturated pixel data of the first pixel data based on the second pixel data, and High dynamic range (HDR) image data is output based on the second pixel data and the third pixel data. The portion of the first pixel data has a higher intensity value than the second pixel data.

2. The image sensor according to claim 1, wherein, The first pixel data corresponds to the first exposure time, and the second pixel data corresponds to the second exposure time. The first exposure time is longer than the second exposure time.

3. The image sensor according to claim 1, wherein, The first pixel data corresponds to the first conversion gain, and the second pixel data corresponds to the second conversion gain. The first conversion gain is higher than the second conversion gain.

4. The image sensor according to claim 3, wherein: The pixel array is configured to output a first pixel signal and a second pixel signal based on the exposure of the plurality of pixels within the same time period, and The readout circuit is configured as follows: The first pixel data is output based on the first pixel signal, and The second pixel data is output based on the second pixel signal.

5. The image sensor according to claim 3, wherein, The pixel array is configured as follows: Output a first pixel signal and a first reset signal associated with the first pixel signal, and Output a second pixel signal and a second reset signal associated with the second pixel signal, and The readout circuit is configured as follows: The first pixel data is output based on the first pixel signal and the first reset signal, and The second pixel data is output based on the second pixel signal and the second reset signal.

6. The image sensor according to claim 1, wherein: Each of the first pixel data and the second pixel data is pixel data having a mosaic pattern with a color filter array based on the plurality of pixels, and The image signal processor is also configured to: Based on the first pixel data and the second pixel data, at least one color ratio is obtained between multiple color channels corresponding to the color filter array, and The third pixel data is generated by recovering at least a portion of the saturated pixel data of the first pixel data based on the at least one color ratio and the second pixel data.

7. The image sensor according to claim 6, wherein, The at least one color is different for each of the saturated pixel data of the plurality of first pixel data.

8. The image sensor according to claim 7, wherein: The plurality of color channels includes a first color channel, a second color channel, and a third color channel. The saturated pixel data of the plurality of first pixel data includes the first saturated pixel data of the first sub-pixel. The first sub-pixel is the pixel associated with the first color channel. The image signal processor is further configured to recover the pixel data of the first sub-pixel based on at least one of a first color ratio and a second color ratio corresponding to the first sub-pixel. The first color ratio is the ratio of the first color channel to the second color channel, and the second color ratio is the ratio of the first color channel to the third color channel.

9. The image sensor according to claim 8, wherein, The image signal processor is also configured to: The first pixel data is de-mosaiced to generate pixel data of a second sub-pixel associated with the second color channel and pixel data of a third sub-pixel associated with the third color channel; as well as The pixel data of the first sub-pixel is recovered based on at least one of a first value reflecting the pixel data of the first color ratio and the second sub-pixel, and a second value reflecting the pixel data of the second color ratio and the third sub-pixel.

10. The image sensor according to claim 1, wherein, The image signal processor is also configured to: The fourth pixel data is generated by demosaicing at least a portion of the first pixel data; The fifth pixel data is generated by demosaicing at least a portion of the second pixel data; At least one color ratio is obtained based on the fourth pixel data and the fifth pixel data; as well as The third pixel data is generated by recovering a portion of the saturated pixel data, at least the first pixel data, based on the fourth pixel data and the at least one color ratio. Wherein, the at least one color ratio represents the relationship between the multiple color channels constituting the first pixel data.

11. The image sensor according to claim 1, wherein, The image signal processor is also configured to: Depixelation of HDR image data, and Output de-pixelated HDR image data.

12. The image sensor according to claim 1, wherein: The image signal processor includes a pre-merging unit, which is configured to: Receive the first pixel data and the second pixel data, and Output the fourth pixel data merged with the first pixel data, and Each of the first pixel data and the second pixel data is pixel data having a mosaic pattern with a color filter array based on the plurality of pixels.

13. The image sensor according to claim 12, wherein, The image signal processor includes a compensation unit, which is configured to: Receive the fourth pixel data, and The output is the third pixel data in which at least a portion of the saturated pixel data of the fourth pixel data is recovered.

14. The image sensor according to claim 13, wherein, The compensation unit includes: The color ratio block is configured based on at least one color ratio between the fourth pixel data output and the plurality of color channels corresponding to the color filter array, and A saturation compensation block is configured to output a portion of the saturated pixel data in which at least the fourth pixel data is recovered based on the at least one color ratio and the third pixel data of the fourth pixel data.

15. The image sensor according to claim 14, wherein, The color block is configured as follows: Receive the fourth pixel data, The fifth pixel data is obtained by de-mosaicing and blurring the fourth pixel data, and The at least one color ratio is obtained based on the fifth pixel data.

16. The image sensor according to claim 14, wherein: The saturation compensation block is also configured to: Receive the at least one color ratio from the color ratio block, the at least one color ratio including a first color ratio and a second color ratio, and The first sub-pixel is recovered based on at least one of the first color ratio and the second color ratio, and at least one of the second sub-pixel and the third sub-pixel. The first sub-pixel, the second sub-pixel, and the third sub-pixel correspond to the saturated pixels in the plurality of color channels obtained by de-mosaicing the first pixel data. The plurality of color channels includes a first color channel, a second color channel, and a third color channel, and the at least one color ratio includes the first color ratio and the second color ratio. The first sub-pixel is a pixel associated with the first color channel, and the first color channel is a color channel associated with the saturated pixel data. The first color ratio is the ratio of the first color channel to the second color channel, and the second color ratio is the ratio of the first color channel to the third color channel.

17. An electronic device comprising: An image sensor is configured to output image data based on pixel signals output from multiple pixels; and A processor is configured to receive the image data and output an image based on the image data to a display device or store the image data in a storage device, wherein: The image sensor includes: The readout circuit is configured to output first pixel data and second pixel data based on the pixel signal; and The image signal processor is configured as follows: The third pixel data is generated by recovering at least a portion of the saturated pixel data of the first pixel data based on the second pixel data, and High dynamic range (HDR) image data is output based on the second pixel data and the third pixel data. The portion of the first pixel data has a higher intensity value than the second pixel data.

18. The electronic device according to claim 17, wherein, The image sensor also includes: The pixel array is configured as follows: Based on the exposure of the multiple pixels within the same time period, a first reset signal is output, followed by a first pixel signal, and a second pixel signal, followed by a second reset signal. The readout circuit is further configured as follows: The first pixel data is output based on the first pixel signal and the first reset signal, and The second pixel data is output based on the second pixel signal and the second reset signal.

19. The electronic device according to claim 17, wherein: Each of the first pixel data and the second pixel data is pixel data having a mosaic pattern with a color filter array based on the plurality of pixels, and The image signal processor is also configured to: Based on the first pixel data and the second pixel data, at least one color ratio is obtained between multiple color channels corresponding to the color filter array, and The third pixel data is generated by recovering at least a portion of the saturated pixel data of the first pixel data based on the at least one color ratio and the second pixel data.

20. A method of operating an image sensor, the method comprising: The pixel array, comprising multiple pixels, outputs pixel signals and a reset signal. The readout circuit outputs first pixel data and second pixel data based on the pixel signal and the reset signal; The third pixel data is generated by recovering at least a portion of the saturated pixel data of the first pixel data based on the color ratio between the color channels of the first pixel data; as well as High dynamic range (HDR) image data is output based on the second pixel data and the third pixel data. The portion of the first pixel data has a higher intensity value than the second pixel data.

Images