Image sensor and image processing apparatus

Abstract

To provide e an image sensor for improving image quality by compensating for full pixel values of a saturation pixel group, and an image processing apparatus.SOLUTION: An image sensor includes an image signal processor that performs image processing on full pixel values generated based on pixel signals of all pixels respectively included in a plurality of pixel groups. The image signal processor generates a color ratio of a saturation pixel group by using a target sub pixel value corresponding to the saturation pixel group having a saturated full pixel value, and compensates for the full pixel value of the saturation pixel group based on the color ratio of the saturation pixel group.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image sensor, and more particularly to an image sensor that performs reliable High Dynamic Range (HDR) processing by compensating full pixel values ​​of a group of saturated pixels using sub-pixel values. [Background technology]

[0002] An image sensor is a device that captures a two-dimensional or three-dimensional image of an object. An image sensor generates image data of the object using photoelectric conversion elements that respond to the intensity of light reflected from the object.

[0003] Recently, with the development of CMOS (Complementary Metal-Oxide Semiconductor) technology, CMOS image sensors using CMOS are widely used. When image data contains saturated pixel values, the saturated pixel values ​​are distorted and the image quality is degraded.

[0004] This calls for techniques to compensate for the pixel values ​​of saturated pixels with more accurate pixel values ​​to improve image quality. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] U.S. Pat. No. 1,061,6518 Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been made in consideration of the above-mentioned conventional problems, and an object of the present invention is to provide an image sensor and an image processing device that improve image quality by compensating for full pixel values ​​of saturated pixel groups. [Means for solving the problem]

[0007] In order to achieve the above object, an image sensor according to one aspect of the present invention includes a pixel array including a plurality of pixel groups, each including a plurality of pixels; a readout circuit that outputs a full pixel value generated based on pixel signals of all pixels included in each of the plurality of pixel groups and outputs sub-pixel values ​​generated based on pixel signals of some pixels in each of the plurality of pixel groups; and an image signal processor that performs image processing on the full pixel values, wherein the image signal processor generates a color ratio of the saturated pixel group using a target sub-pixel value corresponding to a saturated pixel group in which the full pixel value is saturated, and compensates the full pixel value of the saturated pixel group based on the color ratio of the saturated pixel group.

[0008] According to another aspect of the present invention, there is provided an image sensor comprising: a pixel array including a plurality of pixel groups, each including a plurality of pixels; a readout circuit for reading out pixel signals output from the pixel array to generate image data; and an image signal processor for image processing the image data, wherein the image data includes full image data obtained by adding up pixel signals generated from all pixels in each of the plurality of pixel groups on a pixel group basis, and sub-image data obtained by adding up pixel signals generated from some pixels in each of the plurality of pixel groups on a pixel group basis, and the image signal processor generates a color ratio for each of the plurality of pixel groups using the sub-image data, detects saturated pixel groups from among the plurality of pixel groups using the full image data, and compensates the full image data corresponding to the saturated pixel groups based on the color ratio corresponding to the saturated pixel groups.

[0009] In order to achieve the above object, an image processing device according to one aspect of the present invention includes an interface that receives full pixel values ​​generated based on pixel signals from an image sensor including a plurality of pixel groups, each including a plurality of pixels; and a processor that detects a saturated pixel group from among the plurality of pixel groups using the full pixel values, generates a color ratio for the saturated pixel group based on target sub-pixel values ​​generated from the image sensor based on pixel signals of some pixels included in the saturated pixel group, detects an unsaturated color component from among a plurality of color components corresponding to the full pixel value of the saturated pixel group, and compensates for the full pixel value of the saturated pixel group using the color ratio for the unsaturated color component. Effect of the Invention

[0010] According to the image sensor and image processing device of the present invention, a color ratio of a saturated pixel group is generated using a sub-pixel value, and a full pixel value of the saturated pixel group is compensated based on the color ratio of the saturated pixel group, thereby compensating the full pixel value of the saturated pixel group to a more accurate value and improving image quality. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram illustrating an image sensor according to one embodiment of the present invention. [Diagram 2] FIG. 2 illustrates a pixel array according to one embodiment of the present invention. [Figure 3A] 2 is a diagram illustrating a pixel group of a pixel array according to an embodiment of the present invention; [Figure 3B] 2 is a diagram illustrating a pixel group of a pixel array according to an embodiment of the present invention; [Figure 4] 1 is a diagram illustrating an image signal processor according to an embodiment of the present invention; [Diagram 5] FIG. 2 is a block diagram illustrating a saturated pixel detector according to an embodiment of the present invention. [Figure 6] 4 is a flowchart illustrating a method of operating an image sensor according to an embodiment of the present invention. [Figure 7] FIG. 4 is a diagram illustrating saturation of pixel values ​​according to an embodiment of the present invention. [Figure 8] 1 is a diagram illustrating a method for generating a color ratio according to an embodiment of the present invention. [Figure 9] FIG. 13 is a diagram for explaining a method for generating a portion of a color ratio according to an embodiment of the present invention. [Figure 10] 5A and 5B are diagrams for explaining the operation of a saturation compensator according to an embodiment of the present invention. [Figure 11] 5A and 5B are diagrams illustrating a method for determining whether color data is saturated according to an embodiment of the present invention. [Figure 12]4A to 4C are diagrams illustrating a method of generating output image data using color data according to an embodiment of the present invention; [Figure 13] 5A and 5B are diagrams illustrating a method of generating output image data using color data of unsaturated colors according to an embodiment of the present invention; [Figure 14] 4A to 4C are diagrams illustrating a method of generating output image data using sub-pixel values ​​according to an embodiment of the present invention; [Figure 15] 1 is a block diagram illustrating an imaging system according to one embodiment of the present invention. [Figure 16] 1 is a block diagram illustrating an electronic device according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, specific examples of the embodiments of the present invention will be described in detail with reference to the drawings. The same reference numerals are used for the same components in the drawings, and duplicated descriptions thereof will be omitted.

[0013] FIG. 1 is a block diagram illustrating an image sensor according to one embodiment of the present invention.

[0014] The image sensor 10 converts an optical signal of an object incident through an optical lens into image data. The image sensor 10 is, for example, a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor.

[0015] The image sensor 10 is mounted in an electronic device having an image or light sensing function. For example, the image sensor 10 may be realized as a personal computer (PC), an Internet of Things (IoT) device, or a portable electronic device. The portable electronic device may include a laptop computer, a mobile phone, a smartphone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital video camera, an audio device, a portable multimedia player (PMP), a personal navigation device (PND), an MP3 player, a handheld game console, an e-book, a wearable device, and the like. The image sensor 10 may be mounted in an electronic device such as a drone or an Advanced Driver Assistance System (ADAS), or an electronic device provided as a component in a vehicle, furniture, manufacturing equipment, a door, various measuring instruments, and the like.

[0016] 1, the image sensor 10 includes a pixel array 100, a row driver 200, a readout circuit 300, a ramp signal generator 400, a timing controller 500, and an image signal processor 600. The readout circuit 300 includes an analog-to-digital conversion circuit 310 (hereinafter, referred to as an ADC circuit) and a data bus 320. In one embodiment, the pixel array 100, the row driver 200, the readout circuit 300, the ramp signal generator 400, the timing controller 500, and the image signal processor 600 may be implemented as one semiconductor chip or semiconductor module. In one embodiment, the pixel array 100, the row driver 200, the readout circuit 300, the ramp signal generator 400, and the timing controller 500 are implemented as one semiconductor chip or semiconductor module, and the image signal processor 600 is implemented as another semiconductor chip or semiconductor module.

[0017] The pixel array 100 includes a number of row lines RL, a number of column lines CL, and a number of pixels PX connected to the row lines RL and the column lines CL and arranged in an array. In one embodiment, the pixels PX are active pixel sensors (APS). The pixel array 100 includes a number of pixels PX that sense light of different wavelengths. The arrangement of the pixels PX may be implemented in various ways. For example, various pixel PX arrangements of the pixel array 100 will be described below with reference to FIGS. 3A and 3B.

[0018] In one embodiment, the pixel array 100 includes pixel groups each including a plurality of pixels PX. For example, the pixel array 100 includes a pixel group in which four pixels PX arranged in two columns and two rows share one microlens. Alternatively, for example, the pixel array 100 includes a pixel group in which two adjacent pixels PX share one microlens. Each of the pixel groups includes a corresponding color filter. A specific configuration of the pixel array 100 will be described later with reference to FIG. 2 and the like.

[0019] Each of the pixels PX includes at least one photoelectric conversion element, and the pixel PX senses light using the photoelectric conversion element and outputs a pixel signal PXS, which is an electrical signal based on the sensed light. For example, the photoelectric conversion element is a light sensing element made of an organic or inorganic material, such as an inorganic photodiode, an organic photodiode, a perovskite photodiode, a phototransistor, a photogate, or a PIN photodiode (pinned photodiode). In one embodiment, each of the pixels PX includes a plurality of photoelectric conversion elements.

[0020] In one embodiment, the pixel signal PXS includes a sub-pixel signal and a full pixel signal. The full pixel signal refers to a pixel signal generated based on pixel signals of all pixels PX included in each pixel group included in the pixel array 100. As an example, the full pixel signal is a signal obtained by adding up pixel signals of all pixels PX included in each pixel group on a pixel group basis. For example, the full pixel signal is a signal output by adding up pixel signals of pixels PX included in a pixel group when the image sensor 10 performs a binning operation.

[0021] The sub-pixel signal refers to a pixel signal generated based on pixel signals of some pixels PX in each pixel group. For example, the sub-pixel signal is a signal obtained by adding up pixel signals of some pixels PX included in each pixel group on a pixel group basis.

[0022] In one embodiment, the sub-pixel signal is a signal generated in order for the image sensor 10 to perform an autofocus function. For example, a pixel group includes a plurality of pixels PX, and some of the pixels PX included in the pixel group are autofocus pixels that perform the autofocus function. The sub-pixel signal is a signal output by adding up pixel signals of the autofocus pixels included in the pixel group. The sub-pixel signal and the full pixel signal are output to the same column line CL, and the sub-pixel signal is output via the column line CL and a separate line.

[0023] A color filter array for transmitting light in a specific spectral region is disposed on the upper portion of the pixels PX, and the color filter disposed on each of the pixels determines the hue sensed by the pixel. However, the present invention is not limited thereto, and in the case of a specific photoelectric conversion element, depending on the embodiment, light in a specific wavelength band may be converted into an electrical signal depending on the level of an electrical signal applied to the photoelectric conversion element.

[0024] In some embodiments, the pixels PX have a multi-layer structure, and each pixel PX includes a plurality of stacked photoelectric conversion elements that convert light in different spectral regions into electrical signals, and electrical signals corresponding to different hues are generated from the plurality of photoelectric conversion elements, i.e., one pixel PX outputs electrical signals corresponding to multiple colors.

[0025] The row driver 200 drives the pixel array 100 in units of rows. The row driver 200 decodes a row control signal (e.g., an address signal) received from the timing controller 500, and selects at least one row line from among the row lines constituting the pixel array 100 in response to the decoded row control signal. For example, the row driver 200 generates a selection signal for selecting one of a plurality of rows. The pixel array 100 then outputs a pixel signal PXS from a row selected by the selection signal provided from the row driver 200.

[0026] The row driver 200 transmits a control signal for outputting the pixel signal PXS to the pixel array 100, and the pixel PX operates in response to the control signal to output the pixel signal PXS. For example, the row driver 200 generates a control signal for controlling the pixel PX to output the pixel signal PXS during the read-out period, and provides the generated control signal to the pixel array 100.

[0027] The readout circuit 300 reads out pixel signals PXS from pixels PX of a row selected by the row driver 200 among the plurality of pixels PX. At this time, the pixel signals PXS include a reset signal or an image signal (or a sensing signal). The readout circuit 300 converts the reset signal and the image signal received from the pixel array 100 through the plurality of column lines CL into a digital signal based on a ramp signal RAMP from the ramp signal generator 400, thereby generating and outputting pixel values ​​pdt corresponding to the plurality of pixels PX in row units.

[0028] The pixel values ​​pdt include full pixel values ​​and sub-pixel values. The full pixel values ​​refer to pixel values ​​obtained by converting full pixel signals received from the pixel array 100 into digital signals. Image data including the full pixel values ​​of each of the pixel groups is referred to as full image data. The sub-pixel values ​​refer to pixel values ​​obtained by converting sub-pixel signals received from the pixel array 100 into digital signals. Image data including the sub-pixel values ​​of each of the pixel groups is referred to as sub-image data. The full pixel values ​​and sub-pixel values ​​of each of the pixel groups are output to the image signal processor 600. The full image data and the sub-image data are transmitted to the image signal processor 600.

[0029] The ADC circuit 310 includes a plurality of ADCs (Analog to Digital Converters) corresponding to the plurality of column lines CL, and each of the plurality of ADCs compares a reset signal and an image signal received through a corresponding column line CL with a ramp signal RAMP, and generates a pixel value pdt based on the comparison result. For example, the ADC removes the reset signal from the image signal and generates a pixel value pdt indicating the amount of light sensed at the pixel PX. The plurality of pixel values ​​pdt generated by the ADC circuit 310 are output via a data bus 320.

[0030] The ADC circuit 310 includes a plurality of CDS circuits (not shown) (Correlated Double Sampling circuits) and a plurality of counter circuits (not shown). The ADC circuit 310 converts the pixel signal PXS input from the pixel array 100 into a pixel value pdt, which is a digital signal. Each pixel signal PXS is converted into a pixel value pdt, which is a digital signal, by the CDS circuit and the counter circuit.

[0031] The CDS circuit compares the pixel signal PXS with a ramp signal and outputs a comparison result. When the level of the ramp signal and the level of the pixel signal PXS are the same, the CDS circuit outputs a comparison signal that transitions from a first level (e.g., logic high) to a second level (e.g., logic low). The point in time at which the level of the comparison signal transitions is determined by the level of the pixel signal PXS.

[0032] The CDS circuit samples and holds the pixel signal PXS provided from the pixel PX using a correlated double sampling (CDS) method, double samples the level of a specific noise (e.g., a reset signal) and the level due to the image signal, and generates a comparison signal based on the level corresponding to the difference between them.

[0033] The data bus 320 temporarily stores and then outputs the pixel values ​​pdt output from the ADC circuit 310. The data bus 320 includes a plurality of column memories and a column decoder. The pixel values ​​stored in the plurality of column memories are output to an image signal processor 600 in the image sensor 10 under the control of the column decoder.

[0034] The image signal processor 600 receives pixel values ​​pdt. The image signal processor 600 performs image processing operations on the pixel values ​​pdt. The image processing operations include high dynamic range (HDR) processing. The image signal processor 600 performs HDR processing on the pixel values ​​pdt to generate output image data OIDT. The image signal processor 600 receives sub-pixel values ​​and full pixel values. The image signal processor 600 performs HDR processing to compensate for full pixel values ​​of saturated pixel groups.

[0035] In one embodiment, the image signal processor 600 generates a color ratio. As an example, the image signal processor 600 generates a color ratio of a saturated pixel group using a target sub-pixel value. The saturated pixel group refers to a pixel group having a saturated pixel value among the pixel groups, and the target sub-pixel value refers to a sub-pixel value corresponding to the saturated pixel group and generated from the saturated pixel group.

[0036] The image signal processor 600 converts the target sub-pixel value into sub-color data for each of the multiple colors of the saturated pixel group. That is, the target sub-pixel value of the representative color is converted into sub-color data for each of the multiple colors. The image signal processor 600 generates a color ratio based on the sub-color data for each of the multiple colors. For example, the image signal processor 600 converts the target sub-pixel value into sub-color data for the representative color, sub-color data for the first color, and sub-color data for the second color to generate color ratios for the representative color, the first color, and the second color. The image signal processor 600 generates a color ratio between the representative color and the first color, a color ratio between the representative color and the second color, and a color ratio between the first color and the second color.

[0037] The image signal processor 600 generates output image data OIDT by compensating for the full pixel values ​​of the saturated pixel group based on the color ratio of the saturated pixel group. The image signal processor 600 detects unsaturated color components from among color components corresponding to the full pixel values ​​of the saturated pixel group. As an example, the image signal processor 600 converts the full pixel values ​​of the saturated pixel group into color data for each of a plurality of colors. That is, the full pixel value of the representative color is converted into color data for each of a plurality of colors. The image signal processor 600 detects the unsaturated color components by determining whether the color data for each of the plurality of colors is saturated.

[0038] The image signal processor 600 compensates the full pixel values ​​of the saturated pixel group based on the color ratio to the unsaturated color. For example, if the unsaturated color component is a first color, the image signal processor 600 compensates the full pixel values ​​of the saturated pixel group based on the color ratio between the representative color and the first color. If there is no unsaturated color component, the image signal processor 600 compensates the full pixel values ​​of the saturated pixel group based on the target sub-pixel value.

[0039] The image sensor generates a color ratio of the saturated pixel group using the sub-pixel value, and compensates the full pixel value of the saturated pixel group based on the color ratio of the saturated pixel group, thereby compensating the full pixel value of the saturated pixel group as a more accurate value, thereby performing high dynamic range (HDR) processing with improved reliability.

[0040] FIG. 2 is a diagram illustrating a pixel array according to one embodiment of the present invention.

[0041] 2, the pixel array 100 includes a plurality of pixel groups PG each including two or more pixels PX adjacent to each other. As an example, each of the pixel groups PG includes four pixels PX. However, this is not necessarily the case, and each of the pixel groups PG may include various numbers of pixels PX, such as 2, 9, 16, etc. For example, the pixel array 100 includes a plurality of pixel groups PG each including pixels PX arranged in an n×n matrix (n is a positive integer). However, the present invention is not limited thereto, and the pixel array 100 may include a plurality of pixel groups PG each including pixels PX arranged in a 2n×2n matrix (n is a positive integer).

[0042] The full pixel signal is a signal obtained by summing pixel signals of all pixels PX included in each pixel group PG on a pixel group basis. For example, when the image sensor 10 performs a binning operation, pixel signals of the pixels PX included in the pixel group PG are summed to output a full pixel signal.

[0043] The sub-pixel signal is a signal obtained by adding up pixel signals of some pixels PX included in each pixel group PG on a pixel group basis. As an example, the sub-pixel signal is a signal generated when the image sensor 10 performs an autofocus function. For example, when the image sensor 10 performs an autofocus function, pixel signals of some pixels PX included in the pixel group PG are added up to output the sub-pixel signal. However, this is not necessarily the case, and the sub-pixel signal may be output when the image sensor 10 performs a binning operation.

[0044] 3A is a diagram illustrating a pixel group of a pixel array according to an embodiment of the present invention, and in particular, FIG. 3A is a diagram illustrating a pixel array 100a having a tetra pattern, which is an example of a portion of the pixel array 100 of FIG.

[0045] 1 and 3A will be referred to together below. In FIG. 3A, the pixel array 100a has a tetra pattern, but the image sensor 10 can be applied to pixel PX arrays having other patterns.

[0046] The pixel array 100a includes a plurality of pixel groups, for example, first to fourth pixel groups (PG1 to PG4). Each of the plurality of pixel groups (for example, the first to fourth pixel groups (PG1 to PG4)) includes n×n pixels. For example, each of the first to fourth pixel groups (PG1 to PG4) includes four pixels PX arranged in two rows and two columns (2×2). As an example, each of the first to fourth pixel groups (PG1 to PG4) includes one microlens arranged on the four pixels PX. Although FIG. 3A illustrates the pixel array 100a as including four pixel groups, this is merely for convenience of explanation, and the pixel array 100a may include various numbers of pixel groups.

[0047] For example, pixel signals generated by four pixels PX included in one pixel group in which one microlens is arranged differ depending on the shape and refractive index of the microlens, and at least some of the four pixels PX included in one pixel group are pixels PX for an autofocusing (AF) function.

[0048] The pixel array 100a includes color filters to sense various colors. In one embodiment, the same color filters are disposed on the n×n pixels included in each pixel group. As an example, each of the first to fourth pixel groups (PG1 to PG4) includes one of a green (G) color filter, a red (R) color filter, and a blue (B) color filter. In one embodiment, the ratio of the red (R) color filter, the green (G) color filter, and the blue (B) color filter in the pixel array 100a is 1:2:1. As an example, a green (G) color filter is provided on each pixel of the first pixel group PG1. A red (R) color filter is provided on each pixel of the second pixel group PG2. A blue (B) color filter is provided on each pixel of the third pixel group PG3. A green (G) color filter is provided on each pixel of the fourth pixel group PG4.

[0049] In one embodiment, a plurality of pixel groups (e.g., first to fourth pixel groups (PG1 to PG4)) included in the pixel array 100a include the same color filter in pixel group unit. That is, four pixels PX arranged adjacent to each other in the pixel array 100a include color filters of the same color. As an example, the pixel array 100a has a tetra pattern. For example, the pixels PX of the first pixel group PG1 and the fourth pixel group PG4 include green (G) color filters. The pixels PX of the second pixel group PG2 include red (R) color filters. The pixels PX of the third pixel group PG3 include blue (B) color filters. However, this is not necessarily limited thereto, and each of the first to fourth pixel groups (PG1 to PG4) may include at least one of a white color filter, a yellow color filter, a cyan color filter, and a magenta color filter. Alternatively, each of the first to fourth pixel groups (PG1 to PG4) may include one of a white color filter, a yellow color filter, a green (G) color filter, a red (R) color filter, and a blue (B) color filter.

[0050] 1 and 3A, the image sensor 10 generates image data. Specifically, the readout circuit 300 generates image data including pixel values ​​pdt. The image data includes full image data FIDT and sub-image data SIDT. The full image data FIDT includes full pixel values ​​of each of the pixel groups. The full pixel values ​​are generated based on pixel signals of all pixels PX included in each of the pixel groups. As an example, the full pixel values ​​are pixel values ​​generated by adding up pixel signals of all pixels PX included in each of the pixel groups on a pixel group basis.

[0051] The readout circuit 300 generates full image data FIDT based on full pixel signals of the first to fourth pixel groups (PG1 to PG4). The full image data FIDT includes a first full pixel value fpdt1, a second full pixel value fpdt2, a third full pixel value fpdt3, and a fourth full pixel value fpdt4. The first full pixel value fpdt1, the second full pixel value fpdt2, the third full pixel value fpdt3, and the fourth full pixel value fpdt4 correspond to the first pixel group PG1, the second pixel group PG3, the third pixel group PG3, and the fourth pixel group PG4, respectively. The full image data FIDT includes full pixel values ​​of representative colors corresponding to each pixel group. The color of each pixel value included in the image data is referred to as a representative color. For example, since the first pixel group PG1 corresponds to a green color, the first full pixel value fpdt1 is a full pixel value of the green color, which is the representative color. The second pixel group PG2 corresponds to the red color, so the second full pixel value fpdt2 is the full pixel value of the red color, which is the representative color.

[0052] For example, the pixel array 100a outputs a first full pixel signal from the pixels included in the first pixel group PG1. The pixel array 100a outputs the first full pixel signal by summing pixel signals generated from the first pixel PX1, the second pixel PX2, the third pixel PX3, and the fourth pixel PX4 included in the first pixel group PG1. A first full pixel value fpdt1 is generated based on the first full pixel signals generated from the first pixel PX1, the second pixel PX2, the third pixel PX3, and the fourth pixel PX4 included in the first pixel group PG1. In the same manner, a second full pixel value fpdt2, a third full pixel value fpdt3, and a fourth full pixel value fpdt4 are also generated.

[0053] The sub-image data SIDT includes sub-pixel values ​​of each of the pixel groups. The sub-pixel values ​​are generated based on pixel signals of some of the pixels PX included in each of the pixel groups. For example, the sub-pixel values ​​are pixel values ​​generated by adding up pixel signals of some of the pixels PX included in each of the pixel groups on a pixel group basis.

[0054] The readout circuit 300 generates sub-image data SIDT based on sub-pixel signals of the first to fourth pixel groups (PG1 to PG4). The sub-image data SIDT includes a first sub-pixel value spdt1, a second sub-pixel value spdt2, a third sub-pixel value spdt3, and a fourth sub-pixel value spdt4. The first sub-pixel value spdt1, the second sub-pixel value spdt2, the third sub-pixel value spdt3, and the fourth sub-pixel value spdt4 correspond to the first pixel group PG1, the second pixel group PG3, the third pixel group PG3, and the fourth pixel group PG4, respectively. The sub-image data SIDT includes sub-pixel values ​​of representative colors corresponding to each pixel group. For example, since the first pixel group PG1 corresponds to green color, the first sub-pixel value spdt1 is a sub-pixel value of the green color which is the representative color.

[0055] For example, the pixel array 100a outputs a first sub-pixel signal from a portion of pixels included in the first pixel group PG1. For example, pixel signals of one or more pixels PX included in the first pixel group PG1 are excluded from the summation of pixel signals to obtain the first sub-pixel value spdt1. The pixel array 100a outputs the first sub-pixel signal by summing pixel signals generated from the first pixel PX1 and the third pixel PX3 included in the first pixel group PG1. Although FIG. 3A illustrates that sub-pixel signals are generated from two pixels adjacent vertically in one pixel group, this is not necessarily the case, and sub-pixel signals may be generated in various patterns and combinations of the number of pixels. The first sub-pixel value spdt1 is generated based on the first sub-pixel signals generated from the first pixel PX1 and the third pixel PX3 included in the first pixel group PG1. In the same manner, the second sub-pixel value spdt2, the third sub-pixel value spdt3, and the fourth sub-pixel value spdt4 are also generated.

[0056] FIG. 3B is a diagram for explaining a pixel group of a pixel array according to an embodiment of the present invention. In detail, FIG. 3B is a diagram for explaining a pixel array 100b having a hexadeca pattern, which is an example of a portion of the pixel array 100 of FIG. 1. In the following, both FIG. 1 and FIG. 3B will be referred to.

[0057] Each of the first to fourth pixel groups (PG1 to PG4) includes 16 pixels PX arranged in 4 rows and 4 columns (4×4). As an example, in each of the first to fourth pixel groups (PG1 to PG4), microlenses are arranged in units of 4 pixels PX among the 16 pixels PX included in each pixel group. Each of the first to fourth pixel groups (PG1 to PG4) includes 4 microlenses. The pixels PX of the first pixel group PG1 and the fourth pixel group PG4 include green (G) color filters. The pixels PX of the second pixel group PG2 include red (R) color filters. The pixels PX of the third pixel group PG3 include blue (B) color filters. However, this is not necessarily limited thereto.

[0058] The pixel array 100b outputs a first full pixel signal from the pixels included in the first pixel group PG1. The pixel array 100b generates a first full pixel signal by adding up pixel signals generated from the first pixel PX1 to the sixteenth pixel PX16 included in the first pixel group PG1, and generates a first full pixel value fpdt1 based on the first full pixel signal. In the same manner, the second full pixel value fpdt2, the third full pixel value fpdt3, and the fourth full pixel value fpdt4 are also generated.

[0059] For example, the pixel array 100b outputs a first sub-pixel signal from a portion of pixels included in the first pixel group PG1. The pixel array 100b outputs the first sub-pixel signal by adding up pixel signals generated from each of the first pixel PX1 to the eighth pixel PX8 included in the first pixel group PG1. However, this is not necessarily limited thereto, and the sub-pixel signals may be generated in various patterns and combinations of the number of pixels. A first sub-pixel value spdt1 is generated based on the first sub-pixel signal. The second sub-pixel value spdt2, the third sub-pixel value spdt3, and the fourth sub-pixel value spdt4 are also generated in the same manner.

[0060] 4 is a diagram illustrating an image signal processor according to an embodiment of the present invention, and a description that overlaps with the description of FIG.

[0061] 4, the image signal processor 600 receives sub-image data SIDT, which includes sub-pixel values ​​for each of the pixel groups PG, and full image data FIDT, which includes full pixel values ​​for each of the pixel groups PG.

[0062] The image signal processor 600 includes a color ratio generator 610 and a saturation compensator 620. The color ratio generator 610 generates a color ratio based on the sub-image data SIDT. The color ratio means the ratio between a plurality of color components in the image data corresponding to each pixel group PG.

[0063] In one embodiment, the image signal processor 600 generates color ratios for each of a plurality of pixel groups PG. The color ratio generator 610 receives the sub-image data SIDT and generates color ratios for each of the pixel groups PG using the sub-image data SIDT. The color ratio generator 610 generates color ratios for each of the pixel groups PG using sub-pixel values ​​included in the sub-image data SIDT.

[0064] The color ratio generator 610 generates the sub-image data SIDT as sub-color data of each of a plurality of colors through full color estimation. The color ratio generator 610 generates a color ratio for each of the pixel groups PG using the sub-color data of each of the plurality of colors. As an example, the color ratio generator 610 converts the sub-image data SIDT into sub-color data of a representative color, a first color, and a second color by demosaicing. The color ratio generator 610 generates a color ratio for each of the pixel groups PG using the sub-color data corresponding to each of the pixel groups PG. For example, the color ratio generator 610 generates a color ratio between the representative color, the first color, and the second color of each of the pixel groups PG.

[0065] In one embodiment, the image signal processor 600 generates a color ratio of the saturated pixel group. The color ratio generator 610 generates the color ratio of the saturated pixel group using the target sub-pixel value. That is, the color ratio generator 610 is embodied to generate only the color ratio of the saturated pixel group based on the target sub-pixel value. The target sub-pixel value refers to the sub-image data SIDT corresponding to the saturated pixel group.

[0066] The color ratio generator 610 generates sub-color data of each of the multiple colors of the saturated pixel group based on the target sub-pixel value through full color estimation. The color of the target sub-pixel value is referred to as a representative color. The color ratio generator 610 generates sub-color data of each of the representative color, the first color, and the second color based on the sub-pixel value of the representative color. The color ratio generator 610 generates sub-color data of each of the representative color, the first color, and the second color through full color estimation based on the target sub-pixel value and the sub-pixel values ​​of the surrounding pixels of the saturated pixel group. The color ratio generator 610 generates sub-color data of each of the representative color, the first color, and the second color using the sub-pixel values ​​of the target sub-pixel value and the colors different from the representative color of the surrounding pixels. As an example, the color ratio generator 610 converts the target sub-pixel value into sub-color data of each of the representative color, the first color, and the second color by performing demosaicing processing. For example, the color ratio generator 610 generates a color ratio between the representative color, the first color, and the second color of the saturated pixel group.

[0067] The saturation compensator 620 compensates the full image data FIDT based on the color ratio generated by the color ratio generator 610. Specifically, the saturation compensator 620 compensates the full image data FIDT corresponding to the saturated pixel group based on the color ratio of the saturated pixel group. That is, the saturation compensator 620 compensates the full pixel values ​​of the saturated pixel group based on the color ratio of the saturated pixel group. The saturation compensator 620 compensates the full pixel values ​​of the saturated pixel group from the full image data FIDT and outputs them as output image data OIDT.

[0068] The image signal processor 600 detects saturated pixel groups. In one embodiment, the saturation compensator 620 detects saturated pixel groups, although this is not necessarily the case and saturated pixel groups may be detected by other components. Saturated pixel groups are described in more detail below in FIG. 5.

[0069] In one embodiment, when a saturated pixel group is detected, the saturation compensator 620 converts the full image data FIDT corresponding to the saturated pixel group into color data for each of the multiple colors to determine whether the color data for each of the multiple colors is saturated. The saturation compensator 620 converts the full pixel values ​​of the saturated pixel group into color data for each of the multiple colors. The color of the full pixel value of the saturated pixel group is the representative color. For example, the saturation compensator 620 converts the full pixel values ​​of the saturated pixel group into color data for each of the representative color, the first color, and the second color.

[0070] The saturation compensator 620 determines which color data of the representative color, the first color, and the second color of the saturated pixel group is saturated. In one embodiment, the saturation compensator 620 determines whether the color data of the first color and the second color of the saturated pixel group is saturated. Since a pixel group whose full pixel value is saturated in the full image data FIDT corresponds to a saturated pixel group, the saturation compensator 620 does not determine whether the color data of the representative color is saturated.

[0071] In one embodiment, the saturation compensator 620 compensates the full pixel values ​​of the saturated pixel group based on the color ratio for the unsaturated color with the color data of the saturated pixel group. For example, if the first color data of the saturated pixel group is not saturated, the saturation compensator 620 receives the color ratio for the first color from the color ratio generator 610 and compensates the full pixel values ​​of the saturated group pixels based on the color ratio for the first color.

[0072] In one embodiment, when the saturation compensator 620 determines whether the color data of each of the plurality of colors is saturated, the color ratio generator 610 generates only color ratios for unsaturated colors. For example, color ratios corresponding to saturated colors are not generated. The saturation compensator 620 compensates full pixel values ​​of the saturated pixel group based on color ratios for unsaturated colors, so the color ratio generator 610 generates only color ratios for unsaturated colors. The color ratio generator 610 generates color ratios for unsaturated colors determined through color data using a target subpixel value. For example, the saturation compensator 620 determines that the first color is saturated using color data of a first color and color data of a second color. The color ratio generator generates a color ratio between the representative color and the second color using the target subpixel value.

[0073] The sub-image data SIDT includes image information of a relatively bright area with lower sensitivity than the full image data FIDT. Therefore, even if an area (or pixel group) is saturated in the full image data FIDT, it is not saturated in the sub-image data SIDT, and a color ratio for the area is obtained using the sub-image data SIDT. The image sensor 10 can perform high dynamic image processing with improved reliability and accuracy by compensating for the saturated area using the color ratio for the saturated area. In addition, the sub-image data SIDT is read out for an autofocus function, so that the saturated area can be compensated for at low cost.

[0074] 5 is a block diagram illustrating a saturated pixel detector according to an embodiment of the present invention, and a description thereof will be omitted.

[0075] Referring to Figure 5, the image signal processor 600 includes a color ratio generator 610 and a saturation compensator 620. The image signal processor 600 further includes a saturated pixel detector 630. The saturated pixel detector 630 detects saturated pixel groups from among the pixel groups included in the pixel array. A saturated pixel group refers to a pixel group that generates a saturated full pixel value from among the full pixel values ​​generated in each of the pixel groups. Although the saturated pixel detector 630 is illustrated separately from the color ratio generator 610 and the saturation compensator 620 in Figure 5, this is not necessarily the case and the saturated pixel detector 630 may be included in the saturation compensator 620 and the color ratio generator 610.

[0076] In one embodiment, the saturated pixel detector 630 detects pixel groups that generate full pixel values ​​equal to or greater than a threshold among the respective full pixel values ​​of the plurality of pixel groups as saturated pixel groups. A method for detecting saturated pixel groups is described below with reference to FIG.

[0077] 6 is a flow chart illustrating a method of operating an image sensor according to an embodiment of the present invention. Specifically, FIG 6 illustrates a method of operating the image signal processor 600 of FIG 5.

[0078] 6, in step S611, the image signal processor detects a saturated pixel group from among the plurality of pixel groups. The image signal processor receives full pixel values ​​generated based on pixel signals of pixels included in each of the plurality of pixel groups. The image signal processor detects a saturated pixel group from among the plurality of pixel groups using the full pixel values.

[0079] In step S612, the image signal processor generates a color ratio of the saturated pixel group based on the target sub-pixel value, which is a pixel value generated based on pixel signals of some pixels included in the saturated pixel group.

[0080] In step S613, the image signal processor compensates for the full pixel values ​​of the saturated pixel group using a color ratio. The image signal processor uses a color ratio for unsaturated color components among the color components corresponding to the full pixel value. In one embodiment, if there are no unsaturated color components, the image signal processor does not use a color ratio.

[0081] FIG. 7 is a diagram illustrating saturation of pixel values ​​according to an embodiment of the present invention.

[0082] 7, the horizontal axis of the graph in Fig. 7 represents illuminance, and the vertical axis represents pixel value pdt. Hereinafter, Fig. 7 and Fig. 5 will be referred to together.

[0083] 5 and 7, a first graph m1 is a graph showing the full pixel value of a first pixel group PG1 according to illuminance. At a first illuminance L1, the full pixel value of the first pixel group PG1 is a first full pixel value fpdt1. A second graph m2 is a graph showing the full pixel value of a second pixel group PG2 according to illuminance. At a first illuminance L1, the full pixel value of the second pixel group PG2 is a second full pixel value fpdt2.

[0084] The saturated pixel detector 630 detects pixel groups that generate full pixel values ​​equal to or greater than a threshold value among the full pixel values ​​of the pixel groups as saturated pixel groups. The first full pixel value fpdt1 at the first illuminance L1 is equal to or greater than the threshold value, and the second full pixel value fpdt2 at the first illuminance L1 is less than the threshold value. The saturated pixel detector 630 detects the first pixel group PG1 as a saturated pixel group. The saturation compensator 620 compensates the first full pixel value fpdt1.

[0085] FIG. 8 is a diagram illustrating a method for generating a color ratio according to an embodiment of the present invention.

[0086] Specifically, Fig. 8 illustrates the operation of a color ratio generator (e.g., color ratio generator 610 of Fig. 4). For ease of explanation, it is assumed that the color ratio generator generates a color ratio of a saturated pixel group. The description of Fig. 4 will not be repeated.

[0087] In FIG. 8, it is assumed that the first pixel group PG1 is detected as a saturated pixel group. The first sub-pixel value spdt1 of the first pixel group PG1 is a target sub-pixel value. The image signal processor generates a color ratio of the saturated pixel group. The image signal processor generates a color ratio of the first pixel group PG1. The image signal processor generates sub-color data SCIDT for each of the multiple colors of the first pixel group PG1 based on the first sub-pixel value spdt1 through full color estimation. As an example, the full color estimation includes correlation between the multiple colors, demosaicing processing, etc.

[0088] The image signal processor generates sub-color data SCIDT of the representative color, the first color, and the second color based on the first sub-pixel value spdt1 of the representative color. The representative color, the first color, and the second color are different colors. For example, the representative color of the first sub-pixel value spdt1 is green, the first color is red, and the second color is blue. As an example, the image signal processor generates sub-color data SCIDT of red, blue, and green by demosaicing the first sub-pixel value spdt1. The image signal processor generates sub-color data SCIDT of green, red, and blue using the first sub-pixel value spdt1 and the sub-pixel values ​​of the neighboring pixels of the first pixel group PG1.

[0089] The image signal processor generates a color ratio between the representative color, the first color, and the second color of the first pixel group PG1 using the sub-color data SCIDT. The color ratio between the representative color and the first color is a first color ratio CR1, the color ratio between the representative color and the second color is a second color ratio CR2, and the color ratio between the first color and the second color is a third color ratio CR3. As an example, the image signal processor generates the first color ratio CR1 between the green color and the red color using the sub-color data of the green color and the sub-color data of the red color. The image signal processor generates the second color ratio CR2 between the green color and the blue color using the sub-color data of the green color and the sub-color data of the blue color. The image signal processor generates the third color ratio CR3 between the blue color and the red color using the sub-color data of the blue color and the sub-color data of the red color.

[0090] Even if the full pixel value of the saturated pixel group is saturated, the sub-pixel value of the saturated pixel group is not saturated. The image signal processor can obtain the color ratio of the saturated pixel group more accurately using the sub-pixel value of the saturated pixel group. The image sensor of the present invention can compensate the saturated full pixel value as a more accurate pixel value using the color ratio.

[0091] 9 is a diagram for explaining a method of generating a part of a color ratio according to an embodiment of the present invention. Specifically, FIG. 9 illustrates the operation of a color ratio generator (e.g., color ratio generator 610 of FIG. 4). Compared with FIG. 8, FIG. 9 illustrates a case where a color ratio for an unsaturated color is generated. Descriptions that overlap with those of FIG. 8 will be omitted.

[0092] In one embodiment, once the image signal processor (e.g., saturation compensator 620 of FIG. 4) determines whether the color data for each of the multiple colors is saturated, the image signal processor (e.g., color ratio generator 610 of FIG. 4) generates color ratios only for the unsaturated colors.

[0093] The image signal processor generates a color ratio for the unsaturated color using the first sub-pixel value spdt1. As an example, the image signal processor determines the second color as a saturated color, and the image signal processor generates a color ratio between the representative color and the first color using the first sub-pixel value spdt1. For example, the image signal processor generates a first color ratio CR1 between the green color and the red color using the sub-color data of the green color and the sub-color data of the red color.

[0094] FIG. 10 is a diagram for explaining the operation of the saturation compensator according to one embodiment of the present invention.

[0095] The saturation compensator 620 in Fig. 10 corresponds to the saturation compensator 620 in Fig. 4, and therefore a duplicated description will be omitted. In Fig. 10, it is assumed that the first pixel group PG1 is detected as a saturated pixel group.

[0096] 10, the saturation compensator 620 receives the full image data FIDT. The saturation compensator 620 compensates the full image data FIDT based on a color ratio to generate output image data OIDT. The saturation compensator 620 compensates the first full pixel value fpdt1 with a first full pixel value fpdt1' based on a color ratio of the saturated pixel group.

[0097] In one embodiment, the saturation compensator 620 compensates full pixel values ​​of the saturated pixel group and the surrounding pixel group. The surrounding pixel group means pixel groups around the saturated pixel group. For example, the second pixel group PG2, the third pixel group PG3, the fourth pixel group PG4, etc. are the surrounding pixel groups. The saturation compensator 620 compensates full pixel values ​​of each of the surrounding pixel groups using a color ratio of each of the surrounding pixel groups. The image sensor also compensates full pixel values ​​of the surrounding pixel groups using a color ratio, thereby enabling a smooth image representation at the boundary between the saturated pixel group and the surrounding pixel group.

[0098] FIG. 11 is a diagram for explaining a method for determining whether color data is saturated or not according to an embodiment of the present invention.

[0099] Fig. 11 shows the operation of the saturation compensator. It is assumed in Fig. 11 that the first pixel group PG1 is detected as a saturated pixel group. The description overlapping with the contents described above in Fig. 4 will be omitted.

[0100] The saturation compensator generates color data CIDT for each of the multiple colors of the first pixel group PG1 based on the first full pixel value fpdt1 through full color estimation. The saturation compensator generates color data CIDTR for the representative color, color data CIDT1 for the first color, and color data CIDT2 for the second color based on the first full pixel value fpdt1 for the representative color. The representative color, the first color, and the second color are different colors. For example, the representative color of the first full pixel value fpdt1 is a green color, the first color is a red color, and the second color is a blue color. The saturation compensator generates color data through full color estimation based on the first full pixel value fpdt1 and full pixel values ​​of the surrounding pixels of the saturated pixel group. The saturation compensator generates color data for each of the representative color, the first color, and the second color using full pixel values ​​of colors different from the first full pixel value fpdt1 and the representative color of the surrounding pixels. As an example, the saturation compensator demosaices the first full pixel value fpdt1 to generate color data of red color, color data of blue color, and color data of green color.

[0101] In one embodiment, the saturation compensator determines whether the color data CIDT of each of the multiple colors is saturated. The saturation compensator determines whether the color data CIDT of each of the multiple colors corresponding to the first pixel group PG1 is saturated. The saturation compensator determines which color data of the representative color, the color data CIDT1 of the first color, and the color data CIDT2 of the second color is saturated. In one embodiment, the saturation compensator determines whether each of the color data CIDT1 of the first color and the color data CIDT2 of the second color of the first pixel group PG1 is saturated.

[0102] In one embodiment, the saturation compensator compensates the full pixel values ​​of the saturated pixel group based on a color ratio of the saturated pixel group's color data to the unsaturated colors. The saturation compensator uses the unsaturated color data CIDT when compensating the first full pixel value fpdt1. The saturation compensator does not use the saturated color data CIDT when compensating the first full pixel value fpdt1. The saturation compensator does not use the saturated color data CIDT and therefore does not use a color ratio of the saturated color data to the colors.

[0103] FIG. 12 is a diagram illustrating a method of generating output image data using color data according to an embodiment of the present invention.

[0104] In Fig. 12, it is assumed that the first pixel group PG1 is detected as a saturated pixel group. In detail, Fig. 12 assumes that the first color data CIDT1 and the second color data CIDT2 of the first pixel group PG1 are unsaturated. Since the first color data CIDT1 and the second color data CIDT2 are unsaturated, the first color and the second color are unsaturated colors. Descriptions that overlap with the above content will be omitted.

[0105] 8 and 12, the saturation compensator (e.g., the saturation compensator 620 of FIG. 4) compensates the full pixel value of the saturated pixel group based on the color data of the unsaturated color and the color ratio for the unsaturated color. The color ratio for the unsaturated color means one or more color ratios between the unsaturated color and the representative color, and includes a first color ratio CR1 and a second color ratio CR2. For example, the color ratio for the first color means the first color ratio CR1 between the first color and the representative color. The color ratio for the second color means the second color ratio CR2 between the second color and the representative color. The saturation compensator receives the color ratio of the unsaturated color from the color ratio generator (e.g., the color ratio generator 610 of FIG. 4). As an example, since the first color and the second color are unsaturated colors, the saturation compensator receives the first color ratio CR1 and the second color ratio CR2.

[0106] In one embodiment, the saturation compensator generates the output image data OIDT by compensating the first full pixel value fpdt1 based on the color data CIDT1 of the first color, the first color ratio CR1, the color data CIDT2 of the second color, and the second color ratio CR2. As an example, the saturation compensator generates the first full pixel value fpdt1' using the following [Equation 1]. The first full pixel value fpdt1' means a pixel value obtained by compensating the first full pixel value fpdt1. It goes without saying that [Equation 1] is used to compensate full pixel values ​​of other saturated pixel groups in the full image data FIDT.

[0107] [Formula 1] fpdt'=(w1xCIDT1xCR1)+(w2xCIDT2xCR2)

[0108] Here, w1 is a first weighting value and w2 is a second weighting value. The first weighting value is a weighting value for the first color in the first full pixel value fpdt1', and the second weighting value is a weighting value for the second color. As an example, the weighting value is determined by the saturation compensator based on the number of unsaturated colors. If the first color and the second color are unsaturated colors, the first color and the second color are used to generate the first full pixel value fpdt1', so that the first weighting value and the second weighting value are each 0.5.

[0109] The compensated first full pixel value fpdt1' is generated based on the full pixel value of the color component other than the representative color for the first pixel group PG1 and the color ratio with the representative color. Since the representative color is saturated, compensation is performed using the color data and color ratio of the unsaturated color among the color data corresponding to the saturated pixel group. Since compensation is performed using the unsaturated color component in the saturated pixel group and the color ratio with the representative color, the image quality can be improved.

[0110] FIG. 13 is a diagram illustrating a method of generating output image data using color data of unsaturated colors according to an embodiment of the present invention.

[0111] In Fig. 13, it is assumed that the first pixel group PG1 is detected as a saturated pixel group. In detail, Fig. 13 assumes that the color data of the first color of the first pixel group PG1 is not saturated and the second color of the first pixel group PG1 is saturated. Since the color data CIDT1 of the first color is not saturated, the first color is an unsaturated color. Descriptions that overlap with the contents described above in Fig. 12 will be omitted.

[0112] 8 and 13, the saturation compensator receives a first color ratio CR1 from the color ratio generator because the first color is an unsaturated color. In one embodiment, the saturation compensator compensates the full pixel values ​​of the saturated pixel group based on the color data of the unsaturated color and the color ratio for the unsaturated color. As an example, the saturation compensator compensates the first full pixel value fpdt1 based on the color data CIDT1 of the first color and the first color ratio CR1 to generate the output image data OIDT. Because the first color is an unsaturated color, the first color is used to generate the first full pixel value fpdt1'. The second color is not used to generate the first full pixel value fpdt1'. In [Equation 1] above in FIG. 12, the first weight value w1 is 1 and the second weight value w2 is 0.

[0113] FIG. 14 is a diagram illustrating a method of generating output image data using sub-pixel values ​​according to an embodiment of the present invention.

[0114] In Fig. 14, it is assumed that the first pixel group PG1 is detected as a saturated pixel group. In detail, Fig. 14 assumes that the color data of the first color and the color data of the second color of the first pixel group PG1 are saturated. The description overlapping with the above content will be omitted.

[0115] In one embodiment, the saturation compensator compensates the full pixel values ​​of the saturated pixel group using the target sub-pixel value. The saturation compensator compensates the full pixel values ​​of the saturated pixel group using the target sub-pixel value instead of the color ratio because there are no unsaturated colors. As an example, the saturation compensator does not receive the color ratio from the color ratio generator. For example, the saturation compensator compensates the first full pixel value fpdt1 using the first sub-pixel value spdt1, which is the target sub-pixel value, to generate the output image data OIDT.

[0116] In one embodiment, the saturation compensator generates a pixel value obtained by changing the luminance of the target sub-pixel value as the full pixel value of the saturated pixel group. For example, the saturation compensator increases the luminance of the first sub-pixel value spdt1 to generate the first full pixel value fpdt1' of the first pixel group PG1. However, this is not necessarily the case, and if the color data of the first color and the second color are saturated, the saturation compensator may generate the full pixel value of the saturated pixel group using various methods. For example, the saturation compensator maps the first full pixel value pdt1 with a preset pixel value to generate the first full pixel value fpdt1' of the saturated pixel group. In one embodiment, the image signal processor (e.g., the image signal processor 600 of FIG. 4) further performs post-processing for image matching of the full image data FIDT and the sub-image data SIDT.

[0117] The image signal processor determines whether the target sub-pixel value is saturated. In one embodiment, the color ratio generator determines whether the target sub-pixel value is saturated. However, this is not required and a saturation compensator may determine whether the target sub-pixel value is saturated. The color ratio generator discontinues generating a color ratio for the saturated pixel group if the target sub-pixel value is saturated. That is, the color ratio generator does not generate a color ratio for the saturated pixel group if the target sub-pixel value is saturated.

[0118] As an example, if the target subpixel value is saturated, the saturation compensator uses the target subpixel value to compensate the full pixel value of the saturated pixel group. The saturation compensator does not receive a color ratio from the color ratio generator. However, the above method is not necessarily limited, and if the target subpixel value is saturated, the saturation compensator may not use the target subpixel value.

[0119] For example, when the color data of the first and second colors are saturated and the target sub-pixel value is saturated, the saturation compensator maps the first full pixel value fpdt1 as a preset pixel value to generate a first full pixel value fpdt1' of the saturated pixel group. The saturation compensator compensates for the full pixel values ​​of the saturated pixel group using various methods. Also, for example, when the target sub-pixel value is saturated, the saturation compensator may compensate the full image data FIDT using the full pixel value of the saturated pixel group, or may not compensate the full pixel value.

[0120] FIG. 15 is a block diagram illustrating an imaging system according to one embodiment of the present invention.

[0121] Referring to Fig. 15, an image system 1 includes an image sensor 10' and an image processing device 600'. The image processing device 600' corresponds to the image signal processor 600 of Fig. 1. Compared with Fig. 1, the image processing device 600' is provided outside the image sensor 10'. The image processing device 600' includes a color ratio generator 610', a saturation compensator 620', and a saturated pixel detector 630'.

[0122] The image processing device 600' receives the full image data FIDT and the sub-image data SIDT from the image sensor 10'. The description of the image processing device 600', the color ratio generator 610', the saturation compensator 620', and the saturated pixel detector 630' is substantially the same as the description of the image signal processor 600, the color ratio generator 610, the saturation compensator 620, and the saturated pixel detector 630 described above with reference to Figures 1 to 14, and therefore the duplicated description will be omitted.

[0123] 16 is a block diagram showing an electronic device according to an embodiment of the present invention, for example, an electronic device 1000, such as a portable terminal.

[0124] 16, an electronic device 1000 according to an embodiment of the present invention includes an application processor 1200, an image sensor 1100, a display device 1300, a memory 1400, a storage 1500, a user interface 1600, and a wireless transceiver 1700. The description of the image sensor and the method of operating the image sensor according to an embodiment of the present invention described above with reference to FIGS. 1 to 14 are applied to the image sensor 1100.

[0125] The image sensor 1100 generates a color ratio of the saturated pixel group based on the target sub-pixel value, detects an unsaturated color component from among a plurality of color components corresponding to a full pixel value of the saturated pixel group, and compensates the full pixel value of the saturated pixel group using the color ratio for the unsaturated color component.

[0126] The application processor 1200 may be provided as a system-on-chip (SoC) that controls the overall operation of the electronic device 1000 and runs application programs, an operating system (OS), and the like.

[0127] The application processor 1200 receives the output data from the image sensor 1100 .

[0128] The image sensor 1100 generates image data, for example, image data, based on the received optical signal and provides the image data to the application processor 1200. The image data is referred to as pixel values. The image sensor 1100 generates high dynamic range processed image data using color ratios.

[0129] The memory 1400 is implemented as a volatile memory such as a DRAM or an SRAM, or a non-volatile memory such as a FeRAM, a ReRAM, a PRAM, etc. The memory 1400 stores programs and / or data to be processed or executed by the application processor 1200.

[0130] The storage 1500 is embodied as a non-volatile memory device such as a NADN flash or a resistive memory, and for example, the storage 1500 is provided as a memory card (MMC, eMMC, SD, microSD), etc. The storage 1500 stores data and / or programs related to an execution algorithm that controls an image processing operation of the image sensor 1100, and the data and / or programs are loaded into the memory 1400 when the image processing operation is performed. In one embodiment, the storage 1500 stores output image data generated by the image sensor 1100, such as corrected image data or post-processed image data.

[0131] The user interface 1600 may be implemented as a variety of devices for receiving user input, such as a keyboard, a curtain key panel, a touch panel, a fingerprint sensor, a microphone, etc. The user interface 1600 receives the user input and provides a signal corresponding to the received user input to the application processor 1200.

[0132] The wireless transceiver unit 1700 includes a transceiver 1720, a modem 1710, and an antenna 1730.

[0133] Although the embodiments of the present invention have been described in detail above with reference to the drawings, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical spirit of the present invention. [Explanation of symbols]

[0134] 10, 10', 1100 Image Sensor 100, 100a, 100b pixel array 200 Row Driver 300 Readout circuit 310 Analog-to-Digital Conversion (ADC) Circuit 320 Data Bus 400 Ramp Signal Generator 500 Timing Controller 600 Image Signal Processor 600' Image Processing Unit 610, 610' Color Ratio Generator 620, 620' saturation compensator 630, 630' Saturated pixel detector 1000 electronic devices 1200 Application Processor 1300 display device 1400 Memory 1500 Storage 1600 User Interface 1700 Radio Transmitter / Receiver 1710 Modem 1720 Transceiver 1730 Antenna CIDT color data CIDT1, CIDT2 Color data for the first and second colors CIDTR representative color data CL Column Line CR1~CR3 1st~3rd color ratio FIDT full image data fpd1~fpd4 1st~4th full pixel values fpdt1' First full pixel value OIDT Output image data pdt pixel value PG1~PG4 1st~4th pixel group PX Pixels PXS Pixel signal RAMP Ramp signal RL Lowline SCIDT sub-color data SIDT Sub-image data spdt1~spdt4 1st to 4th subpixel values

Claims

1. a pixel array including a plurality of pixel groups, each pixel group including a plurality of pixels; a readout circuit that outputs a full pixel value generated based on pixel signals of all pixels included in each of the plurality of pixel groups, and outputs a sub-pixel value generated based on pixel signals of some pixels in each of the plurality of pixel groups; an image signal processor for image processing the full pixel values; The image signal processor includes: generating a color ratio of the saturated pixel group using a target sub-pixel value corresponding to the saturated pixel group in which the full pixel value is saturated; and compensating full pixel values ​​of the saturated pixel group based on a color ratio of the saturated pixel group.

2. 2. The image sensor of claim 1, wherein the image signal processor generates sub-color data for each of a plurality of colors of the saturated pixel group based on the target sub-pixel value of a representative color through full color estimation, and generates the color ratio based on the sub-color data for each of the plurality of colors.

3. the sub-color data of each of the plurality of colors includes sub-color data of the representative color, a first color, and a second color; The image sensor of claim 2 , wherein the representative color, the first color, and the second color are different from each other.

4. the color ratio includes a plurality of color ratios; The full color estimation includes a demosaicing process; The image signal processor includes: performing the demosaicing process on the target sub-pixel value to generate sub-color data of the representative color, sub-color data of the first color, and sub-color data of the second color; 4. The image sensor according to claim 3, wherein the color ratios are generated for the representative color, the first color, and the second color.

5. The image signal processor includes: converting full pixel values ​​of the saturated pixel group into color data for each of a plurality of colors through full color estimation; 2. The image sensor according to claim 1, further comprising: determining whether or not color data for each of the plurality of colors is saturated.

6. 6. The image sensor of claim 5, wherein the image signal processor compensates full pixel values ​​of the saturated pixel group based on color data of unsaturated colors and the color ratios for the unsaturated colors.

7. The image signal processor includes: converting full pixel values ​​of a representative color for the saturated pixel group into color data of the representative color, a first color, and a second color through full color estimation; 2. The image sensor according to claim 1, wherein it is determined whether the color data of each of the first color and the second color is saturated.

8. 8. The image sensor of claim 7, wherein the image signal processor compensates for full pixel values ​​of the saturated pixel group based on the color data of the first color, the color data of the second color, the color ratio for the first color, and the color ratio for the second color when the color data of each of the first color and the second color is not saturated.

9. 8. The image sensor of claim 7, wherein the image signal processor compensates for full pixel values ​​of the saturated pixel group based on color data of an unsaturated color and a color ratio to the unsaturated color when one of the color data of the first color and the second color is not saturated, based on one of the color data of the first color and the second color being saturated.

10. 8. The image sensor of claim 7, wherein the image signal processor compensates for full pixel values ​​of the saturated pixel group using the target sub-pixel value instead of the color ratio based on color data of each of the saturated first color and the saturated second color.

11. 11. The image sensor of claim 10, wherein the image signal processor generates a pixel value obtained by increasing the luminance of the target sub-pixel value as the full pixel value of the saturated pixel group.

12. 2. The image sensor of claim 1, wherein the image signal processor detects pixel groups having full pixel values ​​equal to or greater than a threshold value from among the full pixel values ​​of the plurality of pixel groups as saturated pixel groups.

13. The image signal processor includes: generating a color ratio for each of the surrounding pixel groups using sub-pixel values ​​for each of the surrounding pixel groups surrounding the saturated pixel group; 2. The image sensor of claim 1, wherein a color ratio of each of the surrounding pixel groups is used to compensate for a full pixel value of each of the surrounding pixel groups.

14. a pixel array including a plurality of pixel groups, each pixel group including a plurality of pixels; a readout circuit for reading out pixel signals output from the pixel array to generate image data; an image signal processor for image processing the image data; the image data includes full image data obtained by summing pixel signals generated from all pixels in each of the plurality of pixel groups on a pixel group basis, and sub-image data obtained by summing pixel signals generated from a portion of pixels in each of the plurality of pixel groups on a pixel group basis, The image signal processor includes: generating a color ratio for each of the plurality of pixel groups using the sub-image data; detecting a saturated pixel group from among the plurality of pixel groups using the full image data; and compensating the full image data corresponding to the saturated pixel group based on the color ratio corresponding to the saturated pixel group.

15. The image signal processor includes: demosaicing the sub-image data to convert it into sub-color data of a representative color, sub-color data of a first color, and sub-color data of a second color; 15. The image sensor of claim 14, further comprising: generating color ratios for the representative color, the first color, and the second color of each of the plurality of pixel groups using the sub-color data of the representative color, the sub-color data of the first color, and the sub-color data of the second color.

16. 15. The image sensor of claim 14, wherein the image signal processor converts the full image data corresponding to the saturated pixel group into color data for each of a plurality of colors based on the detection of the saturated pixel group, and determines whether the color data for each of the plurality of colors is saturated.

17. 17. The image sensor of claim 16, wherein the image signal processor compensates the full image data corresponding to the saturated pixel group based on color data of unsaturated colors and the color ratio for the unsaturated colors.

18. 17. The image sensor of claim 16, wherein the image signal processor compensates the full image data corresponding to the saturated pixel group based on the sub-image data corresponding to the saturated pixel group instead of the color ratio based on the color data of each of the plurality of saturated colors.

19. Each of the plurality of pixel groups includes n×n pixels (n is a positive number); The image sensor according to claim 14 , wherein the same color filter is disposed on the n×n pixels included in each of the plurality of pixel groups.

20. an interface for receiving a full pixel value generated based on the pixel signal from an image sensor including a plurality of pixel groups each including a plurality of pixels; a processor for detecting a saturated pixel group from among the plurality of pixel groups using the full pixel value, generating a color ratio for the saturated pixel group based on a target sub-pixel value generated from the image sensor based on pixel signals of a portion of pixels included in the saturated pixel group, detecting an unsaturated color component from among a plurality of color components corresponding to the full pixel value of the saturated pixel group, and compensating for the full pixel value of the saturated pixel group using the color ratio for the unsaturated color component.

Images