Image sensor and image processing apparatus

Abstract

The invention provides an image sensor and an image processing apparatus. The image sensor includes: a pixel group; a readout circuit configured to: output full-pixel values respectively corresponding to the pixel groups, the full-pixel values being generated based on pixel signals of all pixels in the first pixel group, and sub-pixel values respectively corresponding to the pixel groups, the sub-pixel values being generated based on some of the pixel signals in the first pixel group; and an image signal processor configured to: indicate that the first pixel group is a saturated pixel group based on the full pixel value, generate a color ratio of the saturated pixel group by using a target sub-pixel value corresponding to the saturated pixel group; and compensating the full pixel value of the saturated pixel group based on the color ratio of the saturated pixel group.

Description

[0001] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2023-0143112 filed on October 24, 2023 in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference. Technical Field

[0002] The present disclosure relates to an image sensor, and more particularly, to an image sensor that performs high-reliability high dynamic range (HDR) processing by compensating full pixel values ​​of a saturated pixel group using sub-pixel values. Background Art

[0003] An image sensor is a device that captures a two-dimensional image or a three-dimensional image of an object. The image sensor generates image data of the object by using a photoelectric conversion element that reacts according to the intensity of light reflected or emitted from the object.

[0004] With the recent development of complementary metal oxide semiconductor (CMOS) technology, CMOS image sensors using CMOS are widely used. When saturated pixel values ​​are included in image data, the saturated pixel values ​​may be represented in a distorted manner, and the quality of the image may be deteriorated.

[0005] Therefore, a technique for improving image quality by compensating saturated pixel values ​​using more accurate pixel values ​​is needed. Summary of the invention

[0006] One or more example embodiments provide an image sensor and an image processing apparatus, wherein the image sensor is used to improve image quality by generating a color ratio of a saturated pixel group using sub-pixel values ​​and compensating a full pixel value of the saturated pixel group with a more accurate value based on the color ratio of the saturated pixel group.

[0007] According to one aspect of the example embodiment, an image sensor includes: a pixel array including: a plurality of pixel groups, each of the plurality of pixel groups including a plurality of pixels, the plurality of pixel groups including a first pixel group; a readout circuit configured to: output full pixel values ​​corresponding to the plurality of pixel groups respectively and sub-pixel values ​​corresponding to the plurality of pixel groups respectively, wherein the full pixel value is generated based on pixel signals of all pixels in the first pixel group, and the sub-pixel value is generated based on some of the pixel signals in the first pixel group; and an image signal processor configured to: perform image processing operations based on the full pixel value; indicate that the first pixel group is a saturated pixel group based on the full pixel value, generate a color ratio of the saturated pixel group by using a target sub-pixel value corresponding to the saturated pixel group; and compensate the full pixel value of the saturated pixel group based on the color ratio of the saturated pixel group.

[0008] According to another aspect of the example embodiment, an image sensor includes: a pixel array including: a plurality of pixel groups, each of the plurality of pixel groups including a plurality of pixels, the plurality of pixel groups including a first pixel group; a readout circuit configured to: generate image data based on pixel signals output by the pixel array; and an image signal processor. The image data includes full image data and sub-image data, the full image data being obtained by summing pixel signals generated by all pixels of each of the plurality of pixel groups in units of pixel groups, and the sub-image data being obtained by summing pixel signals generated by some of the pixels of each of the plurality of pixel groups in units of pixel groups. The image signal processor is configured to: perform an image processing operation based on the image data; generate a color ratio of each of the plurality of pixel groups by using the sub-image data; detect the first pixel group as a saturated pixel group based on the full image data; and compensate the full image data corresponding to the saturated pixel group based on the color ratio corresponding to the saturated pixel group.

[0009] According to another aspect of the example embodiment, an image processing device includes: an interface configured to: receive a full pixel value from an image sensor, the full pixel value being generated based on pixel signals of all pixels in each of a plurality of pixel groups of the image sensor, wherein each of the plurality of pixel groups includes a plurality of pixels; and a processor configured to: detect a saturated pixel group among the plurality of pixel groups based on the full pixel value; generate a color ratio of the saturated pixel group based on a target sub-pixel value, the target sub-pixel value being generated based on pixel signals of some pixels in the saturated pixel group; detect an unsaturated color component among a plurality of color components corresponding to the full pixel value of the saturated pixel group; and compensate the full pixel value of the saturated pixel group by using the color ratio of the unsaturated color component. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other aspects and features will become more apparent from the following description of example embodiments taken in conjunction with the accompanying drawings, in which: Figure 1 is a block diagram of an image sensor according to an example embodiment; Figure 2 is a diagram of a pixel array according to an example embodiment; Figure 3A is a diagram of a pixel group of a pixel array according to an example embodiment; Figure 3B is a diagram of a pixel group of a pixel array according to an example embodiment; Figure 4 is a diagram of an image signal processor according to an example embodiment; Figure 5 is a block diagram of an image signal processor including a saturated pixel detector according to an example embodiment; Figure 6is a flowchart of a method of operating an image sensor according to an example embodiment; Figure 7 a graph illustrating saturation of pixel values ​​according to example embodiments; Figure 8 is a diagram of a method of generating a color ratio according to an example embodiment; Fig. 9 is a diagram of a method of generating a portion of a color ratio according to an example embodiment; Fig.10 is a diagram of the operation of a saturation compensator according to an example embodiment; Fig.11 is a diagram of a method of determining whether color data is saturated according to an example embodiment; Fig.12 is a diagram of a method of generating output image data by using color data according to an example embodiment; Fig.13 is a diagram of a method of generating output image data by using color data of a desaturated color according to an example embodiment; Fig.14 is a diagram of a method of generating output image data by using sub-pixel values ​​according to an example embodiment; Fig.15 is a block diagram of an imaging system according to an example embodiment; and Fig.16 is a block diagram of an electronic device according to example embodiments. DETAILED DESCRIPTION

[0011] Hereinafter, example embodiments will be described in detail with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals are used for the same constituent elements, and their repeated description is omitted. The embodiments described herein are example embodiments, so the present disclosure is not limited thereto, and can be implemented in various other forms. Each example embodiment provided in the following description does not exclude being associated with one or more features of other examples or other example embodiments that are also provided herein or not provided herein but consistent with the present disclosure. As used herein, the term "and / or" includes any combination and all combinations of one or more of the associated listed items. When a statement such as "at least one of ..." is after a column of elements, the entire column of elements is modified rather than a single element in the modified column. For example, the statement "at least one of a, b, and c" should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, or all a, b, and c.

[0012] Figure 1 is a block diagram of an image sensor 10 according to an example embodiment.

[0013] The image sensor 10 may convert a light signal of an object incident through an optical lens into image data. The image sensor 10 may include, for example, a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor.

[0014] The image sensor 10 may be mounted on an electronic device having an image sensing function or a light sensing function. For example, the electronic device may be implemented as, for example, a personal computer (PC), an Internet of Things (IoT) device, or a portable electronic device. The portable electronic device may include a laptop computer, a mobile phone, a smart phone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital video camera, an audio device, a portable multimedia player (PMP), a personal navigation device (PND), an MP3 player, a handheld game console, an e-book, a wearable device, and the like. In addition, the image sensor 10 may be mounted as a component in electronic devices (such as drones and advanced driver assistance systems (ADAS)), or vehicles, furniture, manufacturing facilities, doors, various measuring devices, and the like.

[0015] Reference Figure 1 , the image sensor 10 may include a pixel array 100, a row driver 200, a readout circuit 300, a ramp signal generator 400, a timing controller 500, and an image signal processor 600. The readout circuit 300 may include an analog-to-digital conversion (ADC) circuit 310 and a data bus 320. The pixel array 100, the row driver 200, the readout circuit 300, the ramp signal generator 400, the timing controller 500, and the image signal processor 600 may be implemented as a single semiconductor chip or a semiconductor module. The pixel array 100, the row driver 200, the readout circuit 300, the ramp signal generator 400, and the timing controller 500 may include circuits and may be implemented as a single semiconductor chip or a semiconductor module, and the image signal processor 600 may also be implemented as another semiconductor chip or another semiconductor module.

[0016] The pixel array 100 may be connected to a plurality of row lines RL and a plurality of column lines CL, and may include a plurality of pixels PX arranged in an array. In an example embodiment, the plurality of pixels PX may include an active pixel sensor (APS). The pixel array 100 may include a plurality of pixels PX that sense light having different wavelengths. The array of the plurality of pixels PX may be implemented in various ways. For example, the following reference Figure 3A and Figure 3B Various pixel PX arrays of the pixel array 100 are described.

[0017] The pixel array 100 may include pixel groups, each pixel group including a plurality of pixels PX. For example, the pixel array 100 may include a pixel group in which "four pixels PX arranged in two columns and two rows share one microlens". Alternatively, for example, the pixel array 100 may include a pixel group in which "two pixels PX arranged adjacent to each other share one microlens". Each pixel group may include a color filter corresponding thereto. Figure 2 and other figures give a detailed configuration of the pixel array 100 .

[0018] Each of the plurality of pixels PX may include at least one photoelectric conversion element, and the pixel PX may detect light by using the photoelectric conversion element, and output an image signal including an electrical signal according to the sensed light. For example, the photoelectric conversion element may include a light sensing element including an organic material or an inorganic material (such as an inorganic photodiode, an organic photodiode, a perovskite photodiode, a phototransistor, a photogate, a pinned photodiode, etc.). In an example embodiment, each of the plurality of pixels PX may include a plurality of photoelectric conversion elements.

[0019] The pixel signal PXS may include a sub-pixel signal and a full pixel signal. The full pixel signal may represent a pixel signal generated based on the pixel signals of all pixels PX included in each pixel group included in the pixel array 100. For example, the full pixel signal may include a signal obtained by summing the pixel signals of all pixels PX included in each pixel group. For example, the full pixel signal may include a signal obtained by summing the pixel signals of the pixels PX included in the pixel group when the image sensor 10 performs a binning operation.

[0020] The sub-pixel signal may refer to a pixel signal generated based on pixel signals of some pixels PX in each pixel group. For example, the sub-pixel signal may include a signal obtained by summing pixel signals of some pixels PX included in each pixel group.

[0021] The sub-pixel signal may include a signal generated in order to make the image sensor 10 perform an auto focus function. For example, the pixel group may include a plurality of pixels PX, and some of the pixels PX included in the pixel group may include auto focus pixels for performing the auto focus function. The sub-pixel signal may include a signal obtained by summing the pixel signals of the auto focus pixels included in the pixel group. Both the sub-pixel signal and the full-pixel signal may be output to the same column line CL, and the sub-pixel signal may also be output via a line separated from the column line CL.

[0022] A color filter array for transmitting light in a specific spectral range may be arranged on a plurality of pixels PX, and a color detected by a corresponding pixel according to a color filter arranged on each of the plurality of pixels PX may be determined. However, example embodiments are not limited thereto, and in some example embodiments, in the case of a specific photoelectric conversion element, light in a specific wavelength band may also be converted into an electrical signal according to the level of an electrical signal applied to the photoelectric conversion element.

[0023] In some example embodiments, the plurality of pixels PX may have a multilayer structure. The pixel PX having the multilayer structure may include a plurality of stacked photoelectric conversion elements that convert light in different spectral regions into electrical signals, and electrical signals corresponding to different colors may be generated from the plurality of photoelectric conversion elements. In this regard, electrical signals corresponding to a plurality of colors may also be output by one pixel PX.

[0024] The row driver 200 may drive the pixel array 100 in units of rows. The row driver 200 may decode a row control signal (e.g., an address signal) received from the timing controller 500, and in response to the decoded row control signal, the row driver 200 may select at least one row line of the row lines constituting the pixel array 100. For example, the row driver 200 may generate a selection signal for selecting one row among a plurality of rows. In addition, the pixel array 100 may output a pixel signal PXS from a row selected by the selection signal provided by the row driver 200.

[0025] The row driver 200 may send a control signal for outputting the pixel signal PXS to the pixel array 100, and the pixel PX may output the pixel signal PXS by operating in response to the control signal. For example, the row driver 200 may generate a control signal for controlling the pixel PX to output the pixel signal PXS during the readout period, and provide the generated control signal to the pixel array 100.

[0026] The readout circuit 300 may read out a pixel signal PXS from a pixel PX on a row selected by using the row driver 200 among the plurality of pixels PX. In this case, the pixel signal PXS may include a reset signal or an image signal (or a sensing signal). The readout circuit 300 may generate and output pixel values ​​pdt corresponding to the plurality of pixels PX in a row unit by converting the reset signal and the image signal received from the pixel array 100 via the plurality of column lines CL into digital signals based on the ramp signal RAMP from the ramp signal generator 400.

[0027] The pixel value PDT may include a full pixel value and a sub-pixel value. The full pixel value may represent a pixel value of a full pixel signal received from the pixel array 100 that has been converted into a digital signal. Image data including "full pixel values ​​of each pixel group" may be referred to as full image data. Sub-pixel values ​​may represent pixel values ​​of a sub-pixel signal received from the pixel array 100 that has been converted into a digital signal. Image data including "sub-pixel values ​​of each pixel group" may be referred to as sub-image data. The full pixel value and sub-pixel value of each pixel group may be output to the image signal processor 600. The full image data and the sub-image data may be transmitted to the image signal processor 600.

[0028] The ADC circuit 310 may include a plurality of ADCs corresponding to the plurality of column lines CL, respectively, and each of the plurality of ADCs may compare the reset signal and the image signal received via the corresponding column line CL with the ramp signal RAMP, and may generate a pixel value PDT based on the comparison result. For example, the ADC may remove the reset signal from the image signal and generate a pixel value PDT indicating the amount of light sensed by the pixel PX. The plurality of pixel values ​​PDT generated by the ADC circuit 310 may be output via the data bus 320.

[0029] The ADC circuit 310 may include a plurality of correlated double sampling (CDS) circuits and a plurality of counter circuits. The ADC circuit 310 may convert a pixel signal PXS input from the pixel array 100 into a pixel value Pdt as a digital signal. Each pixel signal PXS may be converted into a pixel value Pdt as a digital signal through a CDS circuit and a counter circuit.

[0030] The CDS circuit may compare the pixel signal PXS with the ramp signal and output a comparison result. When the level of the ramp signal RAMP is the same as the level of the pixel signal, the CDS circuit may output a comparison signal that transitions from a first level (e.g., logic high) to a second level (e.g., logic low). The time point of the level transition of the comparison signal may be determined according to the level of the pixel signal PXS.

[0031] The CDS circuit may sample and hold a pixel signal PXS provided by a pixel PX according to a CDS method, double-sample a level of specific noise (eg, a reset signal) and a level according to an image signal, and generate a comparison signal based on a level corresponding to a difference between the levels.

[0032] The data bus 320 may output the pixel value pdt output by the ADC circuit 310 after temporarily storing the pixel value pdt. The data bus 320 may include a plurality of column memories and a column decoder. The plurality of pixel values ​​stored in the plurality of column memories may be output to the image signal processor 600 in the image sensor 10 under the control of the column decoder.

[0033] The image signal processor 600 may receive a pixel value PDT. The image signal processor 600 may perform an image processing operation on the pixel value PDT. The image processing operation may include a high dynamic range (HDR) processing operation. The image signal processor 600 may perform an HDR processing operation on the pixel value PDT and generate output image data OIDT. The image signal processor 600 may receive a sub-pixel value and a full pixel value. The image signal processor 600 may compensate for the full pixel value of the saturated pixel group by performing an HDR processing operation on the pixel value PDT. For example, the image signal processor 600 may perform an image processing operation based on the full pixel value.

[0034] The image signal processor 600 may generate (e.g., may obtain) a color ratio. As an example, the image signal processor 600 may generate a color ratio of each of a plurality of pixel groups by using sub-image data. As an example, the image signal processor 600 may generate a color ratio of a saturated pixel group by using a target sub-pixel value. The saturated pixel group may refer to a pixel group having a saturated pixel value among the pixel groups, and the target sub-pixel value may refer to a sub-pixel value generated by the saturated pixel group as a sub-pixel value corresponding to the saturated pixel group.

[0035] The image signal processor 600 may convert the target sub-pixel value into sub-color data of each color in the plurality of colors of the saturated pixel group. In this regard, the target sub-pixel value of the representative color may be converted into sub-color data of each color in the plurality of colors. The image signal processor 600 may generate a color ratio based on the sub-color data of each color in the plurality of colors. For example, the image signal processor 600 may convert the target sub-pixel value into sub-color data of the representative color, sub-color data of the first color, and sub-color data of the second color, and generate a color ratio between the representative color, the first color, and the second color. The image signal processor 600 may generate a color ratio between the first color and the representative color, a color ratio between the second color and the representative color, and a color ratio between the second color and the first color.

[0036] The image signal processor 600 may generate output image data OIDT by compensating the full pixel value of the saturated pixel group based on the color ratio of the saturated pixel group. The image signal processor 600 may detect an unsaturated color component among the color components corresponding to the full pixel value of the saturated pixel group. As an example, the image signal processor 600 may convert the full pixel value of the saturated pixel group into color data for each color of a plurality of colors. In this regard, the full pixel value of the representative color may be converted into color data for each color of a plurality of colors. The image signal processor 600 may detect the unsaturated color component by determining whether the color data of any color of the plurality of colors is saturated.

[0037] The image signal processor 600 may compensate the full pixel value of the saturated pixel group based on the color ratio of the unsaturated color. For example, when the unsaturated color component is the first color, the image signal processor 600 may compensate the full pixel value of the saturated pixel group based on the color ratio between the representative color and the first color. When there is no unsaturated color component, the image signal processor 600 may compensate the full pixel value of the saturated pixel group based on the target sub-pixel value.

[0038] The image sensor 10 can compensate the full pixel value of the saturated pixel group with a more accurate value by generating a color ratio of the saturated pixel group using sub-pixel values ​​and by compensating the full pixel value of the saturated pixel group based on the color ratio of the saturated pixel group, and can perform HDR processing operations on the pixel values ​​with improved reliability.

[0039] Figure 2 is a diagram of a pixel array 100 according to example embodiments.

[0040] Reference Figure 2 , the pixel array 100 may include a plurality of pixel groups PG, each pixel group PG including two or more pixels PX adjacent to each other. As an example, each pixel group PG may include four pixels PX. However, example embodiments are not necessarily limited thereto, and each pixel group PG may include various numbers of pixels PX, such as 2, 9, and 16. For example, the pixel array 100 may include a plurality of pixel groups PG, each pixel group PG including pixels PX arranged in an n×n matrix (n is a positive integer). However, example embodiments are not limited thereto, and the pixel array 100 may include a plurality of pixel groups PG, each pixel group PG including pixels PX arranged in a 2n×2n matrix (n is a positive integer).

[0041] The full pixel signal may include a signal obtained by summing pixel signals of all pixels PX included in each pixel group PG in units of pixel groups PG. For example, when the image sensor 10 performs a merging operation, the pixel signals of the pixels PX included in the pixel group PG may be summed to be output as a full pixel signal.

[0042] The sub-pixel signal may include a signal obtained by summing pixel signals of some pixels PX included in each pixel group PG in units of pixel groups PG. As an example, the sub-pixel signal may include a signal generated in order for the image sensor 10 to perform an autofocus function. For example, when the image sensor 10 performs an autofocus function action, the pixel signals of some pixels PX included in the pixel group PG may be summed to be output as a sub-pixel signal. However, example embodiments are not necessarily limited thereto, and when the image sensor 10 performs a merging operation, the sub-pixel signal may also be output.

[0043] Figure 3A1 is a diagram of a pixel group of a pixel array 100a according to an example embodiment. The pixel array 100a has a four-in-one (or referred to as a four-in-one pixel, a four-in-one pixel, a tetra) pattern, and is Figure 1 1. Hereinafter, reference will be made to Figure 1 and Figure 3A Although Figure 3A The pixel array 100 a is shown as having a tetra pattern, but the image sensor 10 is applicable to pixel arrays PX having other patterns.

[0044] The pixel array 100a may include a plurality of pixel groups PG (e.g., first pixel groups PG1 to fourth pixel groups PG4). Each of the plurality of pixel groups (e.g., first pixel groups PG1 to fourth pixel groups PG4) may include n×n pixels PX. For example, each of the first pixel groups PG1 to fourth pixel groups PG4 may include four pixels PX arranged in two rows and two columns (i.e., 2×2). As an example, each of the first pixel groups PG1 to fourth pixel groups PG4 may include one microlens arranged on four pixels PX. Although in Figure 3A The pixel array 100 a is illustrated as including four pixel groups PG, but this is merely for convenience of description, and the pixel array 100 a may include various numbers of pixel groups PG.

[0045] As an example, a pixel signal generated by each of four pixels PX included in one pixel group PG on which one microlens is arranged may vary due to the shape and refractive index of the microlens. At least some of the four pixels PX included in one pixel group PG may include pixels PX for an auto focus (AF) function.

[0046] The pixel array 100a may include a color filter for sensing various colors. The same color filter may be arranged on n×n pixels PX included in each pixel group PG. As an example, each of the first pixel group PG1 to the fourth pixel group PG4 may include one of a green G color filter, a red R color filter, and a blue B color filter. In some example embodiments, in the pixel array 100a, the arrangement ratio of the red R color filter, the green G color filter, and the blue B color filter may be 1:2:1. For example, a green G color filter may be disposed on each pixel in the first pixel group PG1. For example, a red R color filter may be disposed on each pixel in the second pixel group PG2. For example, a blue B color filter may be disposed on each pixel in the third pixel group PG3. For example, a green G color filter may be disposed on each pixel in the fourth pixel group PG4.

[0047] The pixels of the plurality of pixel groups PG (e.g., the first pixel group PG1 to the fourth pixel group PG4) included in the pixel array 100a may include the same color filter in units of pixel groups. In this regard, the four pixels PX arranged adjacent to each other in the pixel array 100a may include the same color filter. As an example, the pixel array 100a may have a tetra pattern. For example, the first pixel group PG1 and the fourth pixel group PG4 may include a green G color filter. The second pixel group PG2 may include a red R color filter. The third pixel group PG3 may include a blue B color filter. However, example embodiments are not limited thereto, and each of the first pixel group PG1 to the fourth pixel group PG4 may further include at least one of a white color filter, a yellow color filter, a cyan color filter, and a magenta color filter. Alternatively, each of the first pixel group PG1 to the fourth pixel group PG4 may also include a white color filter, a yellow color filter, a green G color filter, a red R color filter, and a blue B color filter.

[0048] Refer to Figure 1 and Figure 3A , the image sensor 10 may generate image data. The readout circuit 300 may generate image data including pixel values ​​pdt. The image data may include full image data FIDT and sub-image data SIDT. The full image data FIDT may include full pixel values ​​for each pixel group PG. The full pixel value may be generated based on pixel signals of all pixels PX included in each pixel group PG. As an example, the full pixel signal may include a signal obtained by summing pixel signals of all pixels PX included in each of the pixel groups PG.

[0049] The readout circuit 300 may generate full image data FIDT based on full pixel signals of the first pixel group PG1 to the fourth pixel group PG4. The full image data FIDT may include a first full pixel value fpdt1, a second full pixel value fpdt2, a third full pixel value fpdt3, and a fourth full pixel value fpdt4. The first full pixel value fpdt1, the second full pixel value fpdt2, the third full pixel value fpdt3, and the fourth full pixel value fpdt4 may correspond to the first pixel group PG1, the second pixel group PG2, the third pixel group PG3, and the fourth pixel group PG4, respectively. The full image data FIDT may include full pixel values ​​of representative colors corresponding to each pixel group PG. The color of each pixel value included in the image data may be referred to as a representative color. For example, because the first pixel group PG1 corresponds to the green G color, the first full pixel value fpdt1 may be a full pixel value of the green G color as a representative color. Since the second pixel group PG2 corresponds to the red R color, the second full pixel value fpdt2 may be a full pixel value of the red R color as a representative color.

[0050] For example, the pixel array 100a may output a first full pixel signal from a pixel included in the first pixel group PG1. The pixel array 100a may output a first full pixel signal by summing pixel signals generated by each of the first pixel PX1, the second pixel PX2, the third pixel PX3, and the fourth pixel PX4 included in the first pixel group PG1. A first full pixel value fpdt1 may be generated based on the first full pixel signal generated by the first pixel PX1, the second pixel PX2, the third pixel PX3, and the fourth pixel PX4 included in the first pixel group PG1. In this way, a second full pixel value fpdt2, a third full pixel value fpdt3, and a fourth full pixel value fpdt4 may also be generated.

[0051] The sub-image data SIDT may include sub-pixel values ​​of each pixel group PG. The sub-pixel values ​​may be generated based on pixel signals of some pixels PX included in each pixel group PG. As an example, the sub-pixel signal may include a signal obtained by summing pixel signals of some pixels PX included in each pixel group PG.

[0052] The readout circuit 300 may generate sub-image data SIDT based on sub-pixel signals of the first pixel group PG1 to the fourth pixel group PG4. The sub-image data SIDT may include a first sub-pixel value spdt1, a second sub-pixel value spdt2, a third sub-pixel value spdt3, and a fourth sub-pixel value spdt4. The first sub-pixel value spdt1, the second sub-pixel value spdt2, the third sub-pixel value spdt3, and the fourth sub-pixel value spdt4 may correspond to the first pixel group PG1, the second pixel group PG2, the third pixel group PG3, and the fourth pixel group PG4, respectively. The sub-image data SIDT may include sub-pixel values ​​of a representative color corresponding to each pixel group PG. For example, since the first pixel group PG1 corresponds to the green G color, the first sub-pixel value spdt1 may be a sub-pixel value of the green G color as a representative color.

[0053] For example, the pixel array 100a may output the first sub-pixel signal from some pixels included in the first pixel group PG1. For example, the pixel signals of one or more pixels PX included in the first pixel group PG1 may not be used for summing the pixel signals used to obtain the first sub-pixel value spdt1. The pixel array 100a may output the first sub-pixel signal by summing the pixel signals generated by each of the first pixel PX1 and the third pixel PX3 included in the first pixel group PG1. Although Figure 3AThe sub-pixel signal is generated by two vertically adjacent pixels in one pixel group PG, but example embodiments are not necessarily limited thereto, and the sub-pixel signal may also be generated by a combination of various patterns and various numbers of pixels. The first sub-pixel value spdt1 may be generated based on the first sub-pixel signal generated by the first pixel PX1 and the third pixel PX3 included in the first pixel group PG1. In this way, the second sub-pixel value spdt2, the third sub-pixel value spdt3, and the fourth sub-pixel value spdt4 may also be generated.

[0054] Figure 3B 1 is a diagram of a pixel group PG of a pixel array 100b according to an example embodiment. The pixel array 100b has a hexadeca pattern and is Figure 1 1. Hereinafter, reference will be made to Figure 1 and Figure 3B .

[0055] Each of the first pixel group PG1 to the fourth pixel group PG4 may include sixteen pixels PX arranged in four rows and four columns (i.e., 4×4). As an example, in each of the first pixel group PG1 to the fourth pixel group PG4, the microlenses may be arranged in units of four pixels PX among the 16 pixels PX included in each pixel group PG. Each of the first pixel group PG1 to the fourth pixel group PG4 may include four microlenses. The first pixel group PG1 and the fourth pixel group PG4 may include a green G color filter. The second pixel group PG2 may include a red R color filter. The third pixel group PG3 may include a blue B color filter. However, example embodiments are not necessarily limited thereto.

[0056] The pixel array 100b may output a first full pixel signal from a pixel included in the first pixel group PG1. The pixel array 100b may generate a first full pixel signal by summing pixel signals generated by each of the first pixel PX1 to the sixteenth pixel PX16 included in the first pixel group PG1, and may generate a first full pixel value fpdt1 based on the first full pixel signal. In this manner, a second full pixel value fpdt2, a third full pixel value fpdt3, and a fourth full pixel value fpdt4 may also be generated.

[0057] For example, the pixel array 100b may output a first sub-pixel signal from some pixels included in the first pixel group PG1. The pixel array 100b may output a first sub-pixel signal by summing pixel signals generated by each of the first pixel PX1 to the eighth pixel PX8 included in the first pixel group PG1. However, example embodiments are not necessarily limited thereto, and sub-pixel signals may also be generated by using a combination of various patterns and various numbers of pixels. A first sub-pixel value spdt1 may be generated based on the first sub-pixel signal. In this manner, a second sub-pixel value spdt2, a third sub-pixel value spdt3, and a fourth sub-pixel value spdt4 may also be generated.

[0058] Figure 4 is a diagram of an image signal processor 600 according to an example embodiment. Figure 1 Given the duplicate descriptions are omitted.

[0059] Reference Figure 4 , the image signal processor 600 (e.g., an interface (not shown) of the image signal processor 600) may receive sub-image data SIDT and full image data FIDT. The sub-image data SIDT may include sub-pixel values ​​of each pixel group PG, and the full image data FIDT may include full pixel values ​​of each pixel group PG.

[0060] The image signal processor 600 may include a color ratio generator 610 and a saturation compensator 620. The color ratio generator 610 may generate a color ratio based on the sub-image data SIDT. The color ratio may represent an occupation ratio between a plurality of color components in the image data corresponding to each pixel group PG.

[0061] The image signal processor 600 may generate a color ratio for each of the plurality of pixel groups PG. The color ratio generator 610 may receive the sub-image data SIDT and generate a color ratio for each pixel group PG by using the sub-image data SIDT. The color ratio generator 610 may generate a color ratio for each pixel group PG by using a sub-pixel value included in the sub-image data SIDT.

[0062] The color ratio generator 610 may generate sub-image data SIDT as sub-color data for each of a plurality of colors by using full color estimation, and in this regard, the color ratio generator 610 may control the full color estimation operation. The color ratio generator 610 may generate a color ratio for each pixel group PG by using the sub-color data for each of a plurality of colors. As an example, the color ratio generator 610 may perform a demosaicing operation on the sub-image data SIDT, and convert the demosaiced sub-image data SIDT into sub-color data for each of a representative color, a first color, and a second color. The color ratio generator 610 may generate a color ratio for each pixel group PG by using the sub-color data corresponding to each pixel group PG. For example, the color ratio generator 610 may generate a color ratio between a representative color, a first color, and a second color for each pixel group PG.

[0063] The image signal processor 600 may generate a color ratio of a saturated pixel group. The color ratio generator 610 may generate a color ratio of a saturated pixel group by using a target sub-pixel value. In this regard, the color ratio generator 610 may also generate only a color ratio of a saturated pixel group based on the target sub-pixel value. The target sub-pixel value may represent sub-image data SIDT corresponding to the saturated pixel group.

[0064] The color ratio generator 610 can generate sub-color data of each color in a plurality of colors of a saturated pixel group based on a target sub-pixel value by using a full color estimation operation. The color of the target sub-pixel value can be referred to as a representative color. The color ratio generator 610 can generate sub-color data of each of a representative color, a first color, and a second color based on the sub-pixel value of the representative color. The color ratio generator 610 can generate sub-color data based on a target sub-pixel value and sub-pixel values ​​of peripheral pixels of a saturated pixel group by using a full color estimation operation. The color ratio generator 610 can generate sub-color data of each of a representative color, a first color, and a second color by using a target sub-pixel value and sub-pixel values ​​of colors of peripheral pixels different from the representative color. As an example, the color ratio generator 610 can convert the target sub-pixel value into sub-color data of each of a representative color, a first color, and a second color by performing a demosaicing operation on the target sub-pixel value. For example, the color ratio generator 610 can generate a color ratio between a representative color, a first color, and a second color of a saturated pixel group.

[0065] The saturation compensator 620 may compensate the full image data FIDT based on the color ratio generated by the color ratio generator 610. The saturation compensator 620 may compensate the full image data FIDT corresponding to the saturated pixel group based on the color ratio of the saturated pixel group. In this regard, the saturation compensator 620 may compensate the full pixel value of the saturated pixel group based on the color ratio of the saturated pixel group. The saturation compensator 620 may compensate the full pixel value of the saturated pixel group according to the full image data FIDT and output the compensation result as the output image data OIDT.

[0066] The image signal processor 600 may detect a saturated pixel group. The saturation compensator 620 may detect a saturated pixel group. However, example embodiments are not necessarily limited thereto, and a saturated pixel group may be detected by another component. Figure 5 Describe the saturated pixel group in detail.

[0067] When a saturated pixel group is detected, the saturation compensator 620 may convert the full image data FIDT corresponding to the saturated pixel group into color data for each color of a plurality of colors, and may determine whether the color data for any color of the plurality of colors is saturated. The saturation compensator 620 may convert the full pixel value of the saturated pixel group into color data for each color of a plurality of colors. The color of the full pixel value of the saturated pixel group may include a representative color. For example, the saturation compensator 620 may convert the full pixel value of the saturated pixel group into color data for each color of a representative color, a first color, and a second color.

[0068] The saturation compensator 620 may determine which color data is saturated among the color data of each color of the representative color, the first color, and the second color of the saturated pixel group. The saturation compensator 620 may determine whether the color data of each color of the first color and the second color of the saturated pixel group is saturated. Because the pixel group having a saturated full pixel value in the full image data FIDT corresponds to the saturated pixel group, whether the color data of the representative color is saturated may not be determined.

[0069] The saturation compensator 620 may compensate the full pixel value of the saturated pixel group based on the color ratio of the unsaturated color in the color data of the saturated pixel group. For example, when the first color data of the saturated pixel group is unsaturated, the saturation compensator 620 may receive the color ratio of the first color from the color ratio generator 610, and compensate the full pixel value of the saturated pixel group based on the color ratio of the first color.

[0070] When the saturation compensator 620 determines whether the color data of any color among the plurality of colors is saturated, the color ratio generator 610 may also generate only the color ratio of unsaturated colors. For example, the color ratio corresponding to the saturated color may not be generated. Since the saturation compensator 620 compensates the full pixel value of the saturated pixel group based on the color ratio of the unsaturated color, the color ratio generator 610 may also generate only the color ratio of the unsaturated color. The color ratio generator 610 may generate a color ratio of an unsaturated color that has been determined by using the color data and a representative color. For example, the saturation compensator 620 may determine the saturation of the first color by using the color data of the first color and the color data of the second color. The color ratio generator 610 may generate the color ratio between the representative color and the second color by using the target sub-pixel value.

[0071] The sub-image data SIDT has lower sensitivity than the full image data FIDT and may include image information about a relatively bright area. Therefore, a saturated area (or pixel group) in the full image data FIDT may be unsaturated in the sub-image data SIDT, and a color ratio of a corresponding area may be obtained by using the sub-image data SIDT. The image sensor 10 may perform high dynamic image processing with improved reliability and accuracy by compensating for a saturated area using the color ratio of the saturated area. In addition, the sub-image data SIDT may be read out for an autofocus function, etc., and the saturated area may be compensated at a low processing cost.

[0072] Figure 5 is a block diagram of an image signal processor including a saturated pixel detector 630 according to an example embodiment. Duplicated descriptions in the description given above are omitted.

[0073] Reference Figure 5 , the image signal processor 600 may include a color ratio generator 610 and a saturation compensator 620. The image signal processor 600 may further include a saturated pixel detector 630. The saturated pixel detector 630 may detect a saturated pixel group among the pixel groups PG included in the pixel array 100. The saturated pixel group may refer to a pixel group that has generated “a saturated full pixel value among full pixel values ​​generated by each pixel group”. Figure 5 The saturated pixel detector 630 is illustrated as being separated from the color ratio generator 610 and the saturation compensator 620 , but example embodiments are not necessarily limited thereto, and the saturated pixel detector 630 may also be included in the saturation compensator 620 or the color ratio generator 610 .

[0074] The saturated pixel detector 630 may detect a pixel group that generates “a full pixel value greater than or equal to a threshold value among full pixel values ​​of each of a plurality of pixel groups” as a saturated pixel group. Figure 7 A method for detecting groups of saturated pixels is described.

[0075] Figure 6 is a flowchart of a method of operating an image sensor according to example embodiments. Figure 6 Show operation Figure 5 Method of the image signal processor 600.

[0076] Reference Figure 6 In operation S611, the image signal processor may detect a saturated pixel group among the plurality of pixel groups. The image signal processor may receive a full pixel value generated based on a pixel signal of a pixel included in each of the plurality of pixel groups. The image signal processor may detect a saturated pixel group among the plurality of pixel groups by using the full pixel value.

[0077] In operation S612, the image signal processor may generate a color ratio of the saturated pixel group based on the target sub-pixel value. The target sub-pixel value may include a pixel value generated based on pixel values ​​of some pixels included in the saturated pixel group.

[0078] In operation S613, the image signal processor may compensate the full pixel value of the saturated pixel group by using a color ratio. The image signal processor may use a color ratio of an unsaturated color component among color components corresponding to the full pixel value. According to an example embodiment, when there is no unsaturated color component, the image signal processor may not use the color ratio.

[0079] Figure 7 FIG. 4 is a graph showing the saturation of a pixel value pdt according to an example embodiment. Figure 7 , Figure 7 The horizontal axis of the graph indicates brightness, and the vertical axis indicates pixel value pdt. Figure 5 right Figure 7 Give a description.

[0080] Reference Figure 5 and Figure 7 , the first line m1 may represent the full pixel value of the first pixel group PG1 relative to the brightness. At the first brightness L1, the full pixel value of the first pixel group PG1 may be the first full pixel value fpdt1. The second line m2 may represent the full pixel value of the second pixel group PG2 relative to the brightness. At the first brightness L1, the full pixel value of the second pixel group PG2 may be the second full pixel value fpdt2.

[0081] The saturated pixel detector 630 may detect a pixel group that generates “a full pixel value greater than or equal to a threshold value among full pixel values ​​of each of a plurality of pixel groups” as a saturated pixel group. A first full pixel value fpdt1 at a first brightness L1 may be equal to or greater than the threshold value, and a second full pixel value fpdt2 at the first brightness L1 may be less than the threshold value. The saturated pixel detector 630 may detect the first pixel group PG1 as a saturated pixel group. The saturation compensator 620 may compensate for the first full pixel value fpdt1.

[0082] Figure 8 is a diagram of a method of generating a color ratio according to example embodiments. Figure 8 Shows a color ratio generator (e.g., Figure 4 For ease of description, it is assumed that the color ratio generator generates the color ratio of the saturated pixel group. Figure 4 Given the duplicate descriptions are omitted.

[0083] exist Figure 8 , it is assumed that the first pixel group PG1 is detected as a saturated pixel group. The first sub-pixel value spdt1 of the first pixel group PG1 may include a target sub-pixel value. The image signal processor may generate a color ratio of the saturated pixel group. The image signal processor may generate a color ratio of the first pixel group PG1. The image signal processor may generate sub-color data SCIDT for each of a plurality of colors of the first pixel group PG1 based on the first sub-pixel value spdt1 by using a full-color estimation operation. As an example, the full-color estimation operation may include obtaining correlations between a plurality of colors, a demosaicing operation, and the like.

[0084] The image signal processor may generate sub-color data SCIDT of each of the representative color, the first color, and the second color based on the first sub-pixel value spdt1 of the representative color. The representative color, the first color, and the second color may be different from each other. For example, the representative color of the first sub-pixel value spdt1 may include green, the first color may include red, and the second color may include blue. As an example, the image signal processor may perform a demosaicing operation on the first sub-pixel value spdt1 and generate sub-color data SCIDT of red, sub-color data SCIDT of blue, and sub-color data SCIDT of green. The image signal processor may generate sub-color data SCIDT of each of green, red, and blue by using the first sub-pixel value spdt1 and the sub-pixel values ​​of the peripheral pixels of the first pixel group PG1.

[0085] The image signal processor may generate a representative color of the first pixel group PG1, a color ratio between the first color, and the second color by using the sub-color data SCIDT. The image signal processor may generate a color ratio between the representative color and the first color as a first color ratio CR1, a color ratio between the representative color and the second color as a second color ratio CR2, and a color ratio between the first color and the second color as a third color ratio CR3. As an example, the image signal processor may generate a first color ratio CR1 between green and red by using sub-color data of green and sub-color data of red. The image signal processor may generate a second color ratio CR2 between green and blue by using sub-color data of green and sub-color data of blue. The image signal processor may generate a third color ratio CR3 between blue and red by using sub-color data of blue and sub-color data of red.

[0086] Even when the full pixel value of the saturated pixel group is saturated, the sub-pixel value of the saturated pixel group may be unsaturated. The image signal processor may accurately obtain the color ratio of the saturated pixel group by using the sub-pixel value of the saturated pixel group. The image sensor may compensate the full pixel value of the saturated pixel group with an accurate pixel value by using the color ratio.

[0087] Fig. 9 is a diagram of a method of generating a portion of a color ratio according to example embodiments. Fig. 9 Shows a color ratio generator (e.g., Figure 4 The color ratio generator 610 in FIG. Figure 8 In contrast, the case where only a color ratio of unsaturated colors is generated is shown. Figure 8 Given the duplicate descriptions are omitted.

[0088] When an image signal processor (e.g. Figure 4 When the saturation compensator 620 in the image signal processor determines whether the color data of any color of the plurality of colors is saturated, the image signal processor (eg, Figure 4 The color ratio generator 610 in FIG. 6 may also generate color ratios for only unsaturated colors.

[0089] The image signal processor may generate a color ratio of an unsaturated color by using the first sub-pixel value spdt1. As an example, the image signal processor may determine the second color as a saturated color, and the image signal processor may generate a color ratio between the representative color and the first color by using the first sub-pixel value spdt1. For example, the image signal processor may generate a first color ratio CR1 between green and red using sub-color data of green and sub-color data of red.

[0090] Fig.10is a diagram of the operation of a saturation compensator according to an example embodiment. Fig.10 The saturation compensator 620 in Figure 4 The saturation compensator 620 in FIG. 1 corresponds to the saturation compensator 620 in FIG. 1 , so repeated description is omitted. Fig.10 In the example, it is assumed that the first pixel group PG1 is detected as a saturated pixel group.

[0091] Reference Fig.10 The saturation compensator 620 may receive the full image data FIDT. The saturation compensator 620 may compensate the full image data FIDT based on the color ratio and generate the output image data OIDT. The saturation compensator 620 may compensate the first full pixel value fpdt1 to the first full pixel value fpdt1' based on the color ratio of the saturated pixel group.

[0092] The saturation compensator 620 may compensate for the full pixel values ​​of the saturated pixel group and the peripheral pixel group. The peripheral pixel group may refer to a pixel group on the periphery of the saturated pixel group or a pixel group adjacent to the saturated pixel group. For example, the second pixel group PG2, the third pixel group PG3, the fourth pixel group PG4, etc. may be included in the peripheral pixel group. The saturation compensator 620 may compensate for the full pixel value of each peripheral pixel group by using the color ratio of each peripheral pixel group. The image sensor may provide a smooth image at the boundary between the saturated pixel group and the peripheral pixel group by compensating the full pixel value of the peripheral pixel group using the color ratio.

[0093] Fig.11 is a diagram of a method of determining whether color data is saturated according to example embodiments. Fig.11 shows the operation of the saturation compensator. Fig.11 In FIG. 5 , it is assumed that the first pixel group PG1 is detected as a saturated pixel group. Figure 4 Given the duplicate descriptions are omitted.

[0094] The saturation compensator may generate color data CIDT for each of the plurality of colors of the first pixel group PG1 based on the first full pixel value fpdt1 by using a full color estimation operation. The saturation compensator may generate color data CIDTR for a representative color, color data CIDT1 for a first color, and color data CIDT2 for a second color based on the first full pixel value fpdt1 for a representative color. The representative color, the first color, and the second color may be different from each other. For example, the representative color of the first full pixel value fpdt1 may include green, the first color may include red, and the second color may include blue. The saturation compensator may generate color data based on the first full pixel value fpdt1 and full pixel values ​​of peripheral pixels of the saturated pixel group by using a full color estimation. The saturation compensator may generate color data for each of the representative color, the first color, and the second color by using the first full pixel value fpdt1 and full pixel values ​​of colors of peripheral pixels different from the representative color. As an example, the saturation compensator may perform a demosaicing operation on the first full pixel value fpdt1 and generate color data for red, color data for blue, and color data for green.

[0095] The saturation compensator may determine whether the color data CIDT of any color among the plurality of colors is saturated. The saturation compensator may determine whether the color data CIDT of any color among the plurality of colors corresponding to the first pixel group PG1 is saturated. The saturation compensator may determine which color data among the color data CIDTR of the representative color, the color data CIDT1 of the first color, and the color data CIDT2 of the second color is saturated. The saturation compensator may determine whether each of the color data CIDT1 of the first color and the color data CIDT2 of the second color of the first pixel group PG1 is saturated. For example, the saturation compensator may determine whether each of the first color and the second color is saturated based on whether each of the color data CIDT1 of the first color and the color data CIDT2 of the second color is equal to or greater than a predetermined threshold value.

[0096] The saturation compensator may compensate the full pixel value of the saturated pixel group based on the color ratio of the unsaturated color in the color data of the saturated pixel group. When the first full pixel value fpdt1 is compensated, the saturation compensator may use the unsaturated color data CIDT. When the first full pixel value fpdt1 is compensated, the saturation compensator may not use the saturated color data CIDT. Because the saturation compensator does not use the saturated color data CIDT, the saturation compensator may not use the color ratio of the color of the saturated color data.

[0097] Fig.12 is a diagram of a method of generating output image data by using color data according to an example embodiment. Fig.12 In FIG. 1 , it is assumed that the first pixel group PG1 is detected as a saturated pixel group. Fig.12 In the embodiment, it is assumed that each of the color data CIDT1 of the first color and the color data CIDT2 of the second color of the first pixel group PG1 is unsaturated. Since the color data CIDT1 of the first color and the color data CIDT2 of the second color are unsaturated, the first color and the second color may be unsaturated colors. The repeated description in the description given above is omitted.

[0098] Refer to Figure 8 and Fig.12 , saturation compensators (e.g., Figure 4 The saturation compensator 620 in the embodiment may compensate the full pixel value of the saturated pixel group based on the color data of the unsaturated color and the color ratio of the unsaturated color. The color ratio of the unsaturated color may include one or more color ratios between the unsaturated color and the representative color, and may include a first color ratio CR1 and a second color ratio CR2. For example, the color ratio of the first color may represent a first color ratio CR1 between the first color and the representative color. The color ratio of the second color may represent a second color ratio CR2 between the second color and the representative color. The saturation compensator may obtain the full pixel value of the saturated pixel group from ... ratio generator (e.g., Figure 4 The color ratio generator 610 in the embodiment receives the color ratio of the unsaturated color. As an example, because the first color and the second color are unsaturated colors, the saturation compensator may receive the first color ratio CR1 and the second color ratio CR2.

[0099] The saturation compensator may compensate the first full pixel value fpdt1 based on the color data CIDT1 of the first color, the first color ratio CR1, the color data CIDT2 of the second color, and the second color ratio CR2, and may generate the output image data OIDT. As an example, the saturation compensator may generate the first full pixel value fpdt1' by using the following Formula 1. The first full pixel value fpdt1' may represent a pixel value obtained by compensating the first full pixel value fpdt1. Formula 1 may be used to compensate the full pixel values ​​of another saturated pixel group in the full image data FIDT.

[0100] [Formula 1] fpdt'=(w1×CIDT1×CR1)+(w2×CIDT2×CR2) In this case, w1 may be a first weight and w2 may be a second weight. The first weight may include a weight of a first color in the first full pixel value fpdt1', and the second weight may include a weight of a second color. For example, the weight may be determined by a saturation compensator based on the number of unsaturated colors. When the first color and the second color are unsaturated colors (i.e., there are two unsaturated colors), because the first color and the second color may be used to generate the first full pixel value fpdt1', the first weight and the second weight may both be approximately 0.5.

[0101] The compensated first full pixel value fpdt1' may be generated based on a color ratio between the other color components except the representative color of the first pixel group PG1 and the representative color. Since the representative color is saturated, the representative color may be compensated by using the color data and the color ratio of the unsaturated color in the color data corresponding to the saturated pixel group. Compensation may be performed by using the color ratio between the unsaturated color component of the saturated pixel group and the representative color, and thus, the quality of the image may be improved.

[0102] Fig.13 is a diagram of a method of generating output image data by using color data of an unsaturated color according to an example embodiment. Fig.13 In FIG. 1 , it is assumed that the first pixel group PG1 is detected as a saturated pixel group. Fig.13 In FIG. 1 , it is assumed that the color data of the first color of the first pixel group PG1 is unsaturated and the second color of the first pixel group PG1 is saturated. Since the color data CIDT1 of the first color is unsaturated, the first color may be an unsaturated color. Fig.12 Given the duplicate descriptions are omitted.

[0103] Reference Figure 8 and Fig.13 , because the first color is an unsaturated color, the saturation compensator may receive a first color ratio CR1 from the color ratio generator. The saturation compensator may compensate the full pixel value of the saturated pixel group based on the color ratio between the representative color of the saturated pixel group and the unsaturated color. As an example, the saturation compensator may compensate the first full pixel value fpdt1 based on the color data CIDT1 of the first color and the first color ratio CR1, and generate the output image data OIDT. Because the first color is an unsaturated color, the first color may be used to generate the first full pixel value fpdt1'. The second color may not be used to generate the first full pixel value fpdt1'. In the above reference Fig.12 In the described formula 1, the first weight w1 may be approximately 1, and the second weight w2 may be approximately 0.

[0104] Fig.14 is a diagram of a method of generating output image data by using sub-pixel values ​​according to an example embodiment. Fig.14 In FIG. 1 , it is assumed that the first pixel group PG1 is detected as a saturated pixel group. Fig.14 In the above description, it is assumed that the color data of the first color and the color data of the second color of the first pixel group PG1 are not saturated. The repeated description in the description given above is omitted.

[0105] The saturation compensator may compensate the full pixel value of the saturated pixel group using the target sub-pixel value. The saturation compensator may not have an unsaturated color, and thus may compensate the full pixel value of the saturated pixel group by using the target sub-pixel value instead of the color ratio. As an example, the saturation compensator may not receive the color ratio from the color ratio generator. For example, the saturation compensator may compensate the first full pixel value fpdt1 by using the first sub-pixel value spdt1 (i.e., the target sub-pixel value), and generate the output image data OIDT.

[0106] The saturation compensator may generate a pixel value that has been changed according to the brightness of the target sub-pixel value as a full pixel value of the saturated pixel group. For example, the saturation compensator may increase the brightness of the first sub-pixel value spdt1, and generate the increased brightness as the first full pixel value fpdt1' of the first pixel group PG1. For example, the saturation compensator may increase the brightness of the first sub-pixel value spdt1 to the full pixel value of the saturated pixel group to obtain the first full pixel value fpdt1' of the first pixel group PG1. However, example embodiments are not necessarily limited thereto, and when the color data of the first color and the second color are saturated, the saturation compensator may generate a result as the full pixel value of the saturated pixel group by using various methods. For example, the saturation compensator may map the first full pixel value pdt1 to a preset pixel value, and may also generate a result as the first full pixel value fpdt1'. An image signal processor (e.g., Figure 4 The image signal processor 600 in FIG. 6 may also additionally perform post-processing to match the image alignment of the full image data FIDT and the sub-image data SIDT.

[0107] The image signal processor may determine whether the target sub-pixel value is saturated. The color ratio generator may determine whether the target sub-pixel value is saturated. However, example embodiments are not necessarily limited thereto, and the saturation compensator may also determine whether the target sub-pixel value is saturated. When the target sub-pixel value is saturated, the color ratio generator may not generate a color ratio of a saturated pixel group. In this regard, when the target sub-pixel value is saturated, the color ratio generator may not generate a color ratio of a saturated pixel group.

[0108] As an example, when the target sub-pixel value is saturated, the saturation compensator may compensate the full pixel value of the saturated pixel group by using the target sub-pixel value. The saturation compensator may not receive the color ratio from the color ratio generator. However, example embodiments are not necessarily limited thereto, and when the target sub-pixel value is saturated, the saturation compensator may not use the target sub-pixel value.

[0109] For example, when the color data of the first color and the second color are saturated and the target sub-pixel value is saturated, the saturation compensator may map the first full pixel value fpdt1 to a preset pixel value, and may also generate a mapping result as a first full pixel value fpdt1' of the saturated pixel group. The saturation compensator may compensate the full pixel value of the saturated pixel group by using various methods. In addition, for example, when the target sub-pixel value is saturated, the saturation compensator may compensate the full image data FIDT by using the full pixel value of the saturated pixel group, or may not compensate the full pixel value.

[0110] Fig.15 is a block diagram of an image system 1 according to an example embodiment.

[0111] Reference Fig.15 The image system 1 may include an image sensor 10' and an image processing device 600'. The image processing device 600' may be connected to Figure 1 Corresponding to the image signal processor 600 in FIG. Figure 1 In contrast, the image sensor 10' may be external to the image processing device 600'. The image processing device 600' may include a color ratio generator 610', a saturation compensator 620', and a saturated pixel detector 630'.

[0112] The image processing device 600' may receive the full image data FIDT and the sub-image data SIDT from the image sensor 10'. As described and referenced above for the image processing device 600', the color ratio generator 610', the saturation compensator 620', and the saturated pixel detector 630' Figures 1 to 14 The descriptions given of the image signal processor 600, the color ratio generator 610, the saturation compensator 620, and the saturated pixel detector 630 are substantially the same, and thus repeated descriptions thereof are omitted.

[0113] Fig.16 is a block diagram of an electronic device 1000 according to an example embodiment. For example, the electronic device 1000 may include a mobile terminal.

[0114] Reference Fig.16 , the electronic device 1000 according to an example embodiment may include an application processor (AP) 1200, an image sensor 1100, a display device 1300, a memory 1400, a storage device 1500, a user interface 1600, and a wireless transceiver 1700. Figures 1 to 14 The description of the image sensor and the operating method of the image sensor of the described example embodiments may be applied to the image sensor 1100 .

[0115] The image sensor 1100 may generate a color ratio of a saturated pixel group based on the target sub-pixel value. The image sensor 1100 may detect an unsaturated color component among a plurality of color components corresponding to a full pixel value of the saturated pixel group, and may compensate the full pixel value of the saturated pixel group by using the color ratio of the unsaturated color component.

[0116] The AP 1200 may be provided as a System on Chip (SoC) that controls the overall operation of the electronic device 1000 and drives an application program, an operating system, and the like.

[0117] The AP 1200 may receive output data from the image sensor 1100 .

[0118] The image sensor 1100 may generate image data based on the received light signal and provide the image data to the AP 1200. The image data may be referred to as a pixel value. The image sensor 1100 may generate HDR-processed image data by using a color ratio.

[0119] The memory 1400 may be implemented as a volatile memory such as a dynamic random access memory (RAM) (DRAM) and a static RAM (SRAM), or a resistive nonvolatile memory such as a ferroelectric RAM (FeRAM), a resistive RAM (RRAM), and a phase change RAM (PRAM). The memory 1400 may store programs and / or data processed or executed by the AP 1200.

[0120] The storage device 1500 may be implemented as a nonvolatile memory such as a NAND flash memory and a resistive memory, and the storage device 1500 may be provided as, for example, a memory card such as a multimedia card (MMC), an embedded MMC (eMMC), a secure card (SD), and a micro SD), etc. The storage device 1500 may store data and / or a program for executing an algorithm for controlling an image processing operation of the image sensor 1100, and when the image processing operation is performed, the data and / or the program may be loaded into the memory 1400. The storage device 1500 may store output image data generated by the image sensor 1100 such as corrected image data or post-processed image data.

[0121] The user interface 1600 may be implemented as various devices capable of receiving user input, such as a keyboard, a curtain key panel, a touch panel, a fingerprint sensor, and a microphone. The user interface 1600 may receive the user input and provide a signal corresponding to the received user input to the AP 1200 .

[0122] The wireless transceiver 1700 may include a transceiver 1720 , a modem 1710 , and an antenna 1730 .

[0123] In some embodiments, according to example embodiments, Figure 1 , Figure 4 , Figure 5 , Fig.15 and Fig.16 Each component represented by the box shown in can be implemented as various numbers of hardware and / or firmware structures that perform the above-mentioned corresponding functions. For example, at least one of these components may include various hardware components that can perform the corresponding functions through the control of one or more microprocessors or other control devices, and various hardware components that can perform the corresponding functions through the control of one or more microprocessors or other control devices include: digital circuits, programmable or non-programmable logic devices or arrays, application-specific integrated circuits (ASICs), transistors, capacitors, logic gates, or other circuits using direct circuit structures (such as memories, processors, logic circuits, lookup tables, etc.). In addition, at least one of these components may also include a processor (such as a central processing unit (CPU), microprocessor, etc. that performs the corresponding functions), or may be implemented by a processor (such as a central processing unit (CPU), microprocessor, etc. that performs the corresponding functions). The functional aspects of the example embodiments can be implemented in algorithms executed on one or more processors. In addition, the components, elements, modules, or units represented by blocks or processing steps may adopt any number of related technologies for electronic configuration, signal processing and / or control, data processing, etc.

[0124] While the inventive concepts have been particularly shown and described with reference to example embodiments thereof, it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the appended claims.

Claims

1. An image sensor, comprising: A pixel array, comprising: a plurality of pixel groups, each of the plurality of pixel groups comprising a plurality of pixels, the plurality of pixel groups comprising a first pixel group; a readout circuit configured to: output a plurality of full pixel values ​​respectively corresponding to the plurality of pixel groups and a plurality of sub-pixel values ​​respectively corresponding to the plurality of pixel groups, wherein the full pixel values ​​corresponding to a first pixel group are generated based on pixel signals of all pixels in the first pixel group, and the sub-pixel values ​​corresponding to the first pixel group are generated based on pixel signals of some pixels in the first pixel group; and an image signal processor configured to: perform an image processing operation based on the plurality of full pixel values, Among them, the image processing operations include: Indicating that the first pixel group is a saturated pixel group based on a full pixel value corresponding to the first pixel group, generating a color ratio of the saturated pixel group by using a target sub-pixel value corresponding to the saturated pixel group; and The full pixel value of the saturated pixel group is compensated based on the color ratio of the saturated pixel group.

2. The image sensor according to claim 1, wherein: The image signal processor is further configured to generate sub-color data for each of the plurality of colors of the saturated pixel group based on the target sub-pixel value of the representative color by controlling the full color estimation operation, and to generate a color ratio based on the sub-color data for each of the plurality of colors.

3. The image sensor according to claim 2, wherein: The sub-color data of each color of the plurality of colors includes sub-color data of each color of the representative color, the first color, and the second color, and Among them, the representative color, the first color and the second color are different colors.

4. The image sensor according to claim 3, wherein: The color ratio of the saturated pixel group includes a plurality of color ratios, wherein the full color estimation operation includes a demosaicing operation, and Wherein, the image signal processor is further configured as: generating sub-color data of a representative color, sub-color data of a first color, and sub-color data of a second color by performing a demosaicing operation based on a target sub-pixel value, and The plurality of color ratios are generated based on the sub-color data of the representative color, the sub-color data of the first color, and the sub-color data of the second color.

5. The image sensor according to claim 1, wherein: The image signal processor is also configured to: converting full pixel values ​​of the saturated pixel group into color data for each of a plurality of colors using a full color estimation operation; and It is determined whether color data for any color in the plurality of colors is saturated.

6. The image sensor according to claim 5, wherein: One of the plurality of colors is an unsaturated color, and The image signal processor is further configured to compensate the full pixel value of the saturated pixel group based on the color data of the unsaturated color and the color ratio of the unsaturated color.

7. The image sensor according to claim 1, wherein: The image signal processor is also configured to: converting a full pixel value of a representative color of the saturated pixel group into color data for each of the representative color, the first color, and the second color by performing a full color estimation operation; and It is determined whether the color data of each of the first color and the second color is saturated.

8. The image sensor according to claim 7, wherein: The image signal processor is also configured to compensate the full pixel value of the saturated pixel group based on the color data of the first color, the color data of the second color, the color ratio of the first color, and the color ratio of the second color, based on the desaturation of the color data of each of the first color and the second color.

9. The image sensor according to claim 7, wherein: One of the first color and the second color is an unsaturated color, and The image signal processor is further configured to compensate the full pixel value of the saturated pixel group based on the saturation of one color data in the color data of each color in the first color and the second color, and based on the color data of the unsaturated color and the color ratio of the unsaturated color.

10. The image sensor according to claim 7, wherein: The image signal processor is further configured to compensate a full pixel value of the saturated pixel group by using a target sub-pixel value instead of a color ratio of the saturated pixel group based on the color data saturation of each of the first color and the second color.

11. The image sensor according to claim 10, wherein: The image signal processor is further configured to generate a pixel value obtained by increasing the brightness of the target sub-pixel value to a full pixel value of the saturated pixel group.

12. The image sensor according to claim 1, wherein: The image signal processor is further configured to detect the first pixel group as a saturated pixel group based on all pixel values ​​corresponding to the first pixel group being equal to or greater than a threshold value.

13. The image sensor according to any one of claims 1 to 12, wherein: The image signal processor is also configured to: generating a color ratio of each peripheral pixel group by using sub-pixel values ​​of each peripheral pixel group adjacent to the first pixel group; and The full pixel value of each peripheral pixel group is compensated by using the color ratio of each peripheral pixel group.

14. An image sensor, comprising: A pixel array, comprising: a plurality of pixel groups, each of the plurality of pixel groups comprising a plurality of pixels, the plurality of pixel groups comprising a first pixel group; a readout circuit configured to: generate image data based on pixel signals output by the pixel array; and Image Signal Processor, wherein the image data includes full image data based on a signal obtained by summing pixel signals generated by all pixels of each of the plurality of pixel groups in units of pixel groups, and sub-image data based on a signal obtained by summing pixel signals generated by some of the pixels of each of the plurality of pixel groups in units of pixel groups, and Wherein, the image signal processor is configured as: performing image processing operations based on the image data, Among them, the image processing operations include: generating a color ratio for each of the plurality of pixel groups by using the sub-image data; detecting the first pixel group as a saturated pixel group based on full image data corresponding to the first pixel group; and Full image data corresponding to the saturated pixel group is compensated based on the color ratio corresponding to the saturated pixel group.

15. The image sensor according to claim 14, wherein: The color ratio of each of the plurality of pixel groups includes a plurality of color ratios, and Wherein, the image signal processor is further configured as: performing a demosaicing operation based on the sub-image data to convert the demosaiced sub-image data into sub-color data of a representative color, sub-color data of a first color, and sub-color data of a second color; and The plurality of color ratios are generated based on the representative color, the first color, and the second color of each of the plurality of pixel groups by using the sub-color data of the representative color, the sub-color data of the first color, and the sub-color data of the second color.

16. The image sensor according to claim 14, wherein: The image signal processor is further configured to: based on the saturated pixel group being detected, convert the full image data corresponding to the saturated pixel group into color data for each of the plurality of colors, and determine whether the color data for each of the plurality of colors is saturated.

17. The image sensor according to claim 16, wherein: The image signal processor is further configured to compensate the full image data corresponding to the saturated pixel group based on the color data of the unsaturated color and the color ratio of the unsaturated color.

18. The image sensor according to claim 16, wherein: The image signal processor is further configured to compensate full image data corresponding to the saturated pixel group based on the sub-image data corresponding to the saturated pixel group instead of the color ratio corresponding to the saturated pixel group based on the color data saturation of each of the plurality of colors.

19. The image sensor according to any one of claims 14 to 18, wherein: Each of the plurality of pixel groups includes n×n pixels, where n is a positive integer, and Here, the same color filter is arranged on n×n pixels included in each of the plurality of pixel groups.

20. An image processing device, comprising: An interface configured to: receive a plurality of full pixel values ​​from an image sensor, the plurality of full pixel values ​​being generated based on pixel signals of all pixels in each of a plurality of pixel groups of the image sensor, wherein each of the plurality of pixel groups respectively includes a plurality of pixels; and The processor is configured as: detecting a saturated pixel group among the plurality of pixel groups based on the plurality of full pixel values; generating a color ratio of the saturated pixel group based on a target sub-pixel value, the target sub-pixel value being generated based on pixel signals of some pixels in the saturated pixel group; detecting an unsaturated color component among a plurality of color components corresponding to a full pixel value of a saturated pixel group; and The full pixel values ​​of the saturated pixel group are compensated by using the color ratios of the unsaturated color components.

Images