Image sensing device and method of operation thereof

Abstract

To provide a technique relating to an electronic apparatus.SOLUTION: An image sensing device 10 performing with improved grid noise cancellation according to the present technique may include: an image sensor 100 obtaining an image using a plurality of pixels; and an image processor 200 configured to use pixel values included in a region 830 of interest included in the image to generate a gain table 920 including gain table values corresponding to a first resolution, convert the gain table 920 into target tables 1010, 1020 including target table values corresponding to a second resolution which is lower than the first resolution, and cancel noise included in the image using the target tables 1010, 1020.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to electronic devices, and more particularly to image sensing devices and methods of operating the same. [Background technology]

[0002] An image sensor is a device that captures images using the properties of semiconductors that react to light. Recently, with the development of the computer and communications industries, there has been an increasing demand for improved image sensors in a variety of fields, including smartphones, digital cameras, game consoles, the Internet of Things, robots, security cameras, and medical micro cameras.

[0003] Image sensors can be broadly divided into CCD (Charge Coupled Device) image sensors and CMOS (Complementary Metal Oxide Semiconductor) image sensors. CCD image sensors have less noise and superior image quality compared to CMOS image sensors. Meanwhile, CMOS image sensors have a simpler driving method and can be implemented with various scanning methods. In addition, CMOS image sensors can integrate signal processing circuits onto a single chip, making it easy to miniaturize products, consuming very little power, and being compatible with CMOS process technology, resulting in low manufacturing costs. Recently, CMOS image sensing devices have been widely used due to their characteristics that make them more suitable for mobile devices. Summary of the Invention [Problem to be solved by the invention]

[0004] SUMMARY OF THE INVENTION Embodiments of the present invention provide an image sensing device and method of operation that provides improved grid noise removal. [Means for solving the problem]

[0005] An image processing device that controls an image sensor that acquires an image using a plurality of pixels according to an embodiment of the present invention may include: a gain table generation unit that generates a gain table including gain table values ​​corresponding to a first resolution using pixel values ​​included in a region of interest included in the image; a gain table binning unit that converts the gain table into a target table including target table values ​​corresponding to a second resolution; and a calibration execution unit that performs a calibration operation to remove noise included in the image based on the target table.

[0006] An image sensing device according to an embodiment of the present invention may include an image sensor that acquires an image using a plurality of pixels, and an image processor that generates a gain table including gain table values ​​corresponding to a first resolution using pixel values ​​included in a region of interest included in the image, converts the gain table into a target table including target table values ​​corresponding to a second resolution lower than the first resolution, and removes noise included in the image using the target table.

[0007] A method of operating an image processing device according to an embodiment of the present invention may include generating an M×M gain value array corresponding to a region of interest of a 2M×2M pixel value array in an image, generating an N×N target value array, and removing noise from the image, where the M value is K times greater than the N value, and the gain values ​​in the gain value array are obtained by dividing a first statistical representative value by a second statistical representative value. The first statistical representative value may be a pixel value of the same color as a reference pixel value corresponding to each gain value in the region of interest, and the second statistical representative value may be the reference pixel value. The target values ​​in the target value array may be third statistical representative values ​​of gain values ​​in a K×K sub-gain value array corresponding to the target value, and the noise may be removed by multiplying the target value by the pixel value of a K×K sub-image array corresponding to each target value, and the K×K sub-image array may be included in the image. [Effects of the Invention]

[0008] According to the present technology, it is possible to provide an image sensing device that performs improved lattice noise removal. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an image sensing device according to an embodiment of the present invention; [Figure 2] 2 is a diagram for explaining the image sensor of FIG. 1 according to an embodiment of the present invention. [Figure 3] 3 is a diagram for explaining a Bayer pattern of the pixel array of FIG. 2 according to an embodiment of the present invention. FIG. [Figure 4] 3 is a diagram illustrating a Quad Bayer pattern of the pixel array of FIG. 2 according to an embodiment of the present invention. [Figure 5] 3 is a diagram for explaining a nanocell pattern of the pixel array of FIG. 2 according to an embodiment of the present invention. FIG. [Figure 6] 3 is a diagram for explaining a hexadecapattern of the pixel array of FIG. 2 according to an embodiment of the present invention. FIG. [Figure 7] 1 is a block diagram illustrating an image sensing device according to an embodiment of the present invention. [Figure 8] FIG. 10 is a diagram for explaining setting of a region of interest according to an embodiment of the present invention. [Figure 9] FIG. 10 is a diagram illustrating a method for generating a gain table according to an embodiment of the present invention. [Figure 10] 10A and 10B are diagrams illustrating a method for binning a gain table according to an embodiment of the present invention. [Figure 11] 10A and 10B are diagrams for explaining pixels to be excluded when generating a gain table according to an embodiment of the present invention. [Figure 12] 1 is a flowchart illustrating a noise calibration method according to an embodiment of the present invention. [Figure 13]10 is a flowchart illustrating a method for binning a gain table according to an embodiment of the present invention. [Figure 14] FIG. 1 is a block diagram illustrating a computing system including an image sensor according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] With respect to embodiments according to the inventive concepts disclosed in this specification or application, specific structural or functional descriptions are merely examples for describing embodiments according to the inventive concepts, and embodiments according to the inventive concepts may be embodied in various forms and should not be construed as being limited to the embodiments described in this specification or application.

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the technical concept of the present invention.

[0012] FIG. 1 is a diagram illustrating an image sensing device according to an embodiment of the present invention.

[0013] Referring to FIG. 1, an image sensing device 10 may include an image sensor 100 and an image processor 200 .

[0014] The image sensing device 10 may be included in an electronic device, such as a digital camera, a mobile phone, a smartphone, a tablet personal computer (PC), a notebook, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital video camera, a portable multimedia player (PMP), a mobile internet device (MID), a personal computer (PC), a wearable device, or a camera for various purposes (such as a front camera, a rear camera, or a dashcam camera for an automobile).

[0015] The image sensor 100 may be implemented as a CCD image sensor or a CMOS image sensor. The image sensor 100 may generate image data for an object (not shown) input (or captured) through a lens (not shown). The lens (not shown) may include at least one lens forming an optical system.

[0016] The image sensor 100 may include a plurality of pixels. The image sensor 100 can generate a plurality of pixel values ​​DPXs corresponding to a captured image from the plurality of pixels. The plurality of pixel values ​​DPXs generated by the image sensor 100 can be transmitted to the image processor 200. That is, the image sensor 100 can generate a plurality of pixel values ​​DPXs corresponding to a single frame.

[0017] The image processor 200 can control the image sensor 100. Specifically, the image processor 200 can perform processing to improve the image quality of pixel data received from the image sensor 100 and output the processed image data. Here, the processing may include EIS (Electronic Image Stabilization), interpolation, color correction, image quality correction, size adjustment, etc. The image processor 200 is also referred to as an image processing device.

[0018] The image processor 200 can calibrate noise contained in an image based on the system information AG and the plurality of pixel values ​​DPXs. The image may include maze noise. Maze noise may be grid-like noise caused by color imbalance among pixels of the same color among a plurality of pixels. Maze noise may be caused by color disparity among pixels of the same color. In an embodiment of the present invention, the maze noise may be grid-like noise caused by crosstalk among a plurality of pixels.

[0019] The image processor 200 can calculate a calibration gain value for performing a calibration operation to reduce maize noise based on a plurality of pixel values.

[0020] 1, the image processor 200 may be implemented as a chip separate from the image sensor 100. In this case, the image sensor 100 chip and the image processor 200 chip may be implemented in a single package, for example, a multi-chip package. In another embodiment of the present invention, the image processor 200 may be included as part of the image sensor 100 and implemented as a single chip.

[0021] FIG. 2 is a diagram illustrating the image sensor of FIG. 1 according to an embodiment of the present invention.

[0022] Referring to FIG. 2, the image sensor 100 may include a pixel array 110, a row decoder 120, a timing generator 130, and a signal converter 140.

[0023] The pixel array 110 may include a color filter array 111 according to the embodiment, and a photoelectric conversion layer 113 formed under the color filter array 111 and including a plurality of photoelectric conversion elements corresponding to each pixel of the color filter array 111. The pixel array 110 may include a plurality of pixels for outputting color information contained in incident light. Each of the plurality of pixels may output a pixel signal corresponding to the incident light passing through the corresponding color filter array 111.

[0024] The color filter array 111 may include color filters that pass only specific wavelengths (e.g., red, blue, or green) of light incident on each pixel. The color filter array 111 allows pixel data for each pixel to indicate a value corresponding to the intensity of light of a specific wavelength, and each pixel may be referred to as a red pixel R, a blue pixel B, or a green pixel G depending on the type of color filter.

[0025] Specifically, each of the plurality of pixels can accumulate photocharges generated in response to incident light and generate a pixel signal corresponding to the accumulated photocharges. Each pixel may include a photoelectric conversion element (e.g., a photodiode, a phototransistor, a photogate, or a pinned photodiode) that converts a photosignal into an electrical signal and at least one transistor that processes the electrical signal.

[0026] The pixel array 110 may include a plurality of pixels arranged in a row direction and a column direction. The pixel array 110 may generate a plurality of pixel signals VPXs for each row. Each of the plurality of pixel signals VPXs may be an analog type pixel signal VPXs.

[0027] The row decoder 120 is responsive to the address and control signals output from the timing generator 130 to select one row from among a number of rows in which a plurality of pixels are arranged in the pixel array 110 .

[0028] The signal converter 140 can convert a plurality of analog pixel signals VPXs into a plurality of digital pixel values ​​DPXs. The plurality of digital pixel values ​​DPXs can be output in various patterns. In response to a control signal output from the timing generator 130, the signal converter 140 can perform correlated double sampling (CDS) on each signal output from the pixel array 110, analog-to-digital convert each CDS-processed signal, and output each digital signal. Each digital signal may be a signal corresponding to the intensity of the wavelength of incident light passing through the corresponding color filter array 111.

[0029] The signal converter 140 may include a CDS (correlated double sampling) block and an ADC (analog to digital converter) block. The CDS block may sequentially sample and hold a reference signal and a video signal set provided to each of a plurality of columns included in the pixel array 110. That is, the CDS block may sample and hold the levels of the reference signal and the video signal corresponding to each column. The ADC block may output pixel data obtained by converting the correlated double sampling signal for each column output from the CDS block into a digital signal. To this end, the ADC block may include a comparator and a counter corresponding to each column.

[0030] Furthermore, the image sensor 100 according to the embodiment of the present invention may further include an output buffer 150. The output buffer 150 may be implemented as a plurality of buffers that store the digital signals output from the signal converter 140. Specifically, the output buffer 150 may latch and output pixel data for each column provided from the signal converter 140. The output buffer 150 may temporarily store the pixel data output from the signal converter 140 and sequentially output the pixel data under the control of the timing generator 130. According to the embodiment of the present invention, the output buffer 270 may be omitted.

[0031] FIG. 3 is a diagram for explaining a Bayer pattern of the pixel array of FIG. 2 according to an embodiment of the present invention.

[0032] Referring to FIG. 3, the pixel array 110 may be arranged in a predetermined pattern. For example, the pixel array 110 may be arranged in a Bayer pattern. As indicated by the dashed lines in FIG. 3, the Bayer pattern may be configured with 2×2 pixel repeating cells. In each cell, two pixels Gb and Gr having green color filters may be arranged diagonally opposite each other, and one pixel B having a blue color filter and one pixel R having a red color filter may be arranged at the remaining corner. The four pixels B, Gb, Gr, and R are not necessarily limited to the arrangement structure shown in FIG. 3 and may be arranged in various ways based on the Bayer pattern described above.

[0033] FIG. 4 is a diagram illustrating a Quad Bayer pattern of the pixel array of FIG. 2 according to an embodiment of the present invention.

[0034] Referring to FIG. 4, the pixel array 110 may be arranged in a predetermined pattern. For example, the pixel array 110 may be arranged in a quad Bayer pattern. As indicated by the dashed lines in FIG. 4, the quad Bayer pattern may be configured with 4×4 pixel repeating cells. In each cell, eight pixels Gb and Gr having green color filters may be arranged diagonally opposite each other, and four pixels B having blue color filters and four pixels R having red color filters may be arranged at the remaining corners. The 16 pixels B, Gb, Gr, and R are not necessarily limited to the arrangement structure shown in FIG. 4 and may be arranged in various ways based on the above-described quad Bayer pattern.

[0035] FIG. 5 is a diagram for explaining a nonacell pattern of the pixel array of FIG. 2 according to an embodiment of the present invention.

[0036] Referring to FIG. 5, the pixel array 110 may be arranged in a predetermined pattern. For example, the pixel array 110 may be arranged in a nona cell pattern. As indicated by the dashed lines in FIG. 5, the nona cell pattern may be configured with a repeating cell of 6×6 pixels. In each cell, 18 pixels Gb and Gr having green color filters may be arranged diagonally opposite each other, and 9 pixels B having blue color filters and 9 pixels R having red color filters may be arranged at the remaining corners. The 36 pixels B, Gb, Gr, and R are not necessarily limited to the arrangement structure shown in FIG. 5 and may be arranged in various ways based on the nona cell pattern described above.

[0037] FIG. 6 is a diagram illustrating a hexadecapattern of the pixel array of FIG. 2 according to an embodiment of the present invention.

[0038] Referring to FIG. 6, the pixel array 110 may be arranged in a predetermined pattern. For example, the pixel array 110 may be arranged in a hexa-deca pattern. As indicated by the dashed lines in FIG. 6, the hexa-deca pattern may be composed of repeating cells of 8×8 pixels. In each cell, 32 pixels Gb and Gr having green color filters may be arranged diagonally opposite each other, and 16 pixels B having blue color filters and 16 pixels R having red color filters may be arranged at the remaining corners. The 64 pixels B, Gb, Gr, and R are not necessarily limited to the arrangement structure shown in FIG. 6 and may be arranged in various ways based on the above-described hexa-deca pattern.

[0039] FIG. 7 is a block diagram illustrating an image sensing device according to an embodiment of the present invention.

[0040] 7, the image sensing device may include an image sensor 100 and an image processor 200. The image sensor 100, which acquires an image using a plurality of pixels, may transmit a plurality of pixel values ​​to the image processor 200. The image processor 200 may remove noise included in the image based on the plurality of pixel values. The image processor 200 may include a gain table generator 210, a gain table binner 220, a calibration executor 230, and a gain table storage unit 240. The image sensor 100 and the image processor 200 may correspond to the descriptions of FIGS. 1 and 2.

[0041] The gain table generator 210 may generate a gain table including gain table values ​​corresponding to a first resolution using pixel values ​​included in a region of interest included in an image. In an embodiment of the present invention, the gain table generator 210 may generate a gain table used in a calibration operation to reduce noise generated at the maximum resolution of an image based on a plurality of pixel values. The noise may include maze noise. The maze noise may be a grid-like noise generated by differences in pixel values ​​of pixels of the same color among a plurality of pixels. Pixel crosstalk may cause pixel values ​​of pixels of the same color to differ.

[0042] The region of interest may be a region of a predetermined size around one of the intersections of a plurality of grid lines superimposed on the image. The gain table generator 210 may set a plurality of grid lines superimposed on the input image. The gain table generator 210 may set a predetermined region of interest at the intersection of the grid lines. The gain table generator 210 may generate a gain table corresponding to the maximum resolution in the region of interest. The gain table generator 210 may generate a gain map for the entire image based on the generated gain table. Specifically, the gain table generator 210 may extend the gain table into a gain map using linear interpolation, which will be described with reference to FIG. 11.

[0043] The generated gain table may be stored in the gain table storage unit 240. The gain table generator 210 may generate the gain table by excluding pixel values ​​corresponding to pixels for phase detection autofocus (PDAF) from pixel values ​​included in the region of interest.

[0044] The gain table may include multiple gain regions. The gain table values ​​may correspond to the multiple gain regions, respectively. The gain table values ​​for each gain region may be calculated by dividing the average value of pixel values ​​of the same color as pixel values ​​corresponding to the multiple gain regions among pixel values ​​included in the region of interest by the average value of pixel values ​​corresponding to the gain region. The gain table generator 210 can generate a gain table including multiple gain table values.

[0045] The gain table binning unit 220 may convert the gain table into a target table including target table values ​​corresponding to a second resolution. In an embodiment of the present invention, the first resolution may be higher than the second resolution. The first resolution may be the maximum resolution, and the second resolution may be the target resolution. For example, the first resolution may be a resolution corresponding to a hexadeca pattern, and the second resolution may be one of a resolution corresponding to a nonacell pattern, a resolution corresponding to a quad-Bayer pattern, and a resolution corresponding to a Bayer pattern. The first resolution may be a resolution corresponding to a nonacell pattern, and the second resolution may be a resolution corresponding to a quad-Bayer pattern or a Bayer pattern. In high illumination where there is sufficient light, the maximum (full) resolution may be used, but in low illumination where there is insufficient light or in video mode shooting, a binned Bayer pattern may be used.

[0046] The target table may include a plurality of calculation regions. Each of the target table values ​​for each calculation region may be an average value of the gain table values ​​corresponding to the calculation region. The number of gain table values ​​corresponding to the plurality of calculation regions may be determined based on the first resolution and the second resolution. For example, if the first resolution corresponds to a hexadeca pattern and the second resolution corresponds to a quad-Bayer pattern, the number of gain table values ​​corresponding to the plurality of calculation regions may be four.

[0047] In another embodiment of the present invention, each of the target table values ​​in each calculation region may be the median of the gain table values ​​corresponding to the calculation region. The number of gain table values ​​included in each of the plurality of calculation regions may be determined based on the first resolution and the second resolution.

[0048] The gain table binning unit 220 may calculate a target table value that is an average value of gain table values ​​corresponding to multiple calculation regions among the gain table values. In another embodiment of the present invention, the gain table binning unit 220 may calculate a median value of gain table values ​​corresponding to multiple calculation regions among the gain table values. The target table value corresponding to a calculation region may be an average value or median value of the gain table values ​​corresponding to the calculation region. The gain table binning unit 220 may generate a target table including the calculated target table values. In an embodiment of the present invention, the number of gain table values ​​may be a positive integer multiple of the number of target table values.

[0049] In an embodiment of the present invention, the gain table binning unit 220 may determine gain table values ​​corresponding to a plurality of calculation regions. For example, the first resolution may be a resolution corresponding to a hexadeca pattern, and the second resolution may be a resolution corresponding to a quad-Bayer pattern. The gain table binning unit 220 may determine four gain table values ​​corresponding to one of the plurality of calculation regions based on the first and second resolutions. The four gain table values ​​may be of the same color. The four gain table values ​​may be adjacent gain table values.

[0050] The calibration unit 230 may perform a calibration operation to remove noise included in an image based on the target table. The calibration unit 230 may apply the target table values ​​to a plurality of pixel values ​​included in the image. For example, the calibration unit 230 may multiply a plurality of pixel values ​​included in the image by the target table values ​​in a second resolution unit.

[0051] In an embodiment of the present invention, the calibration unit 230 may perform a calibration operation to reduce maize noise generated at the second resolution of the input image based on the target table. As a result of the calibration operation of the calibration unit 230, color deviation between pixels of the same color may be reduced, thereby eliminating grid noise.

[0052] In another embodiment of the present invention, the image sensing device 10 may generate a gain table including gain table values ​​corresponding to a first resolution using pixel values ​​included in a region of interest included in an image. The image sensing device 10 may convert the gain table into a target table including target table values ​​corresponding to a second resolution lower than the first resolution. The image sensing device 10 may remove noise included in the image using the target table. The target table values ​​may be generated using the average or median of at least two or more gain table values.

[0053] FIG. 8 is a diagram for explaining setting of a region of interest according to an embodiment of the present invention.

[0054] Referring to FIG. 8, a gain table used in a calibration operation for a portion of an input image can be generated to derive a gain map for the entire image.

[0055] When calculating the calibration gain values, if the gain values ​​are calculated for all pixels included in the image, the image processing speed may be slow and a large amount of storage space may be required. In order to improve the image processing speed and save storage space, a grid method may be used.

[0056] A plurality of grid lines 810 may be set on the input image 800. The set plurality of grid lines 810 may be overlaid on the image 800. Intersections 820 of the grid lines 810 may be set on the image 800. A predetermined region of interest 830 may be set at the intersections 820. The size of the region of interest 830 may be changed depending on the image.

[0057] In Figure 8, since there can be multiple intersections 820, the location of the region of interest 830 can vary. Once a gain table for the region of interest 830 is generated, the gain table can be expanded into a gain map for the entire image 800 using techniques such as linear interpolation. The setting of the region of interest in Figure 8 may be performed by a gain table generator.

[0058] FIG. 9 is a diagram for explaining a method for generating a gain table according to an embodiment of the present invention.

[0059] In FIG. 9, for convenience of explanation, it is assumed that the first resolution of the input image corresponds to a hexadecapattern and the size of the region of interest 830 is 16×16.

[0060] The gain table generator can divide the region of interest 830 to correspond to the first resolution. Since the maximum resolution of the image corresponds to a hexadecapattern, the region of interest 830 is divided into four sub-regions. Reference numeral 910 in FIG. 9 indicates one of the four sub-regions within the region of interest 830.

[0061] The gain table 920 may include gain table values ​​corresponding to each of a plurality of gain regions. For example, one gain table value may correspond to one gain region 921 among the plurality of gain regions. The gain table value corresponding to one gain region 921 may be a value obtained by dividing the average value of pixel values ​​of the same color as the pixel (any one of 911, 913, 915, and 917) corresponding to one gain region 921 among the pixel values ​​included in the region of interest by the average value of the pixels 911, 913, 915, and 917 corresponding to one gain region 921. In FIG. 9 , the pixel (any one of 911, 913, 915, and 917) corresponding to one gain region 921 is green, so the average value of the green pixel values ​​among the pixels included in the region of interest can be calculated. In another embodiment of the present invention, the gain table value corresponding to one gain region 921 may be the average value of pixel values ​​of the same color as the pixel (any one of 911, 913, 915, 917) corresponding to one gain region 921 among the pixel values ​​included in the region of interest, divided by the median value of the pixels 911, 913, 915, 917 corresponding to one gain region 921.

[0062] The remaining gain table values ​​of the gain table 920 can be calculated in a similar manner. According to an embodiment of the present invention, the color of a pixel included in an image sensor may be any one of green Gr, Gb, red R, or blue B. When generating the gain table, the green Gr and Gb pixels may be pixels of the same color. Deviation between the green Gr and Gb pixels may cause noise. The operation of generating the gain table of FIG. 9 may be performed by a gain table generator.

[0063] FIG. 10 is a diagram illustrating a method for binning a gain table according to an embodiment of the present invention.

[0064] 10, for convenience of explanation, it is assumed that the first resolution corresponds to a hexadeca pattern and the second resolution corresponds to a quad-Bayer pattern or a Bayer pattern. The gain table 920 corresponding to the first resolution can be binned to become target tables 1010 and 1020. The binning method in FIG. 10 can correspond to the description of FIG. 7.

[0065] 10, the gain table binning unit 220 may convert the gain table into a target table including target table values ​​corresponding to a second resolution. The target table may include multiple calculation regions. The gain table values ​​may correspond to the multiple calculation regions. The number of gain table values ​​corresponding to the multiple calculation regions may be determined based on the first resolution and the second resolution. The gain table binning unit 220 may determine gain table values ​​corresponding to the multiple calculation regions.

[0066] The gain table binning unit 220 may calculate a target table value that is an average value of the gain table values ​​corresponding to a plurality of calculation regions among the gain table values. In another embodiment of the present invention, the gain table binning unit 220 may calculate a median value that is located in the middle of the gain table values ​​corresponding to a plurality of calculation regions among the gain table values. The target table value corresponding to a calculation region may be an average value or median value of the gain table values ​​corresponding to the calculation region. The gain table binning unit 220 may generate a target table including the calculated target table values.

[0067] 10, the gain table 920 may be converted into a target table 1010. The second resolution may be a resolution corresponding to a Quad Bayer pattern. The gain table binning unit 220 may determine four gain table values ​​corresponding to one of the plurality of calculation regions based on the first resolution and the second resolution. The four gain table values ​​may be the same color. The four gain table values ​​may be adjacent to each other.

[0068] That is, four gain table values ​​of the same color can be binned. Expressed mathematically, the table value of the target table 1010 is Gr1 = (Gr11 + Gr12 + Gr21 + Gr22) / 4. Similarly, the table value of the target table 1010 is Gr2 = (Gr13 + Gr14 + Gr23 + Gr24) / 4. The remaining table values ​​Gr3, Gr4, R1, R2, R3, R4, B1, B2, B3, B4, Gb1, Gb2, Gb3, and Gb4 of the target table 1010 can be calculated in a similar manner.

[0069] 10, the gain table 920 may be converted into a target table 1020. The second resolution may be a resolution corresponding to a Bayer pattern. The gain table binning unit 220 may determine 16 gain table values ​​corresponding to one of the plurality of calculation regions based on the first resolution and the second resolution. The 16 gain table values ​​may be of the same color. The 16 gain table values ​​may be adjacent to each other.

[0070] That is, 16 gain table values ​​of the same color can be binned. Expressed mathematically, the table value of the target table Gr is (Gr11+Gr12+Gr13+Gr14+Gr21+Gr22+Gr23+Gr24+Gr31+Gr32+Gr33+Gr34+Gr41+Gr42+Gr43+Gr44) / 16. Similarly, the table value of the target table Gb is (Gb11+Gb12+Gb13+Gb14+Gb21+Gb22+Gb23+Gb24+Gb31+Gb32+Gb33+Gb34+Gb41+Gb42+Gb43+Gb44) / 16. Because deviations occur only in the green colors Gr and Gb at a resolution corresponding to the Bayer pattern, the remaining table values ​​R and B of the target table 1020 do not need to be calculated.

[0071] According to an embodiment of the present invention, since the gain table binning unit 220 can bin the gain table corresponding to the first resolution, there is no need to store the gain table corresponding to the second resolution.

[0072] In another embodiment of the present invention, a gain table value corresponding to one calculation region may correspond to another calculation region. That is, one gain table value may correspond to multiple calculation regions. For example, a first resolution may correspond to a hexadeca pattern, and a second resolution may correspond to a nona pattern. That is, 16 gain table values ​​of the same color may be binned into 9 target table values. The nine target table values ​​can be expressed as formulas as follows: (Gr11+Gr12+Gr21+Gr22) / 4, (Gr12+Gr13+Gr22+Gr23) / 4, (Gr13+Gr14+Gr23+Gr24) / 4, (Gr21+Gr22+Gr31+Gr32) / 4, (Gr22+Gr23+Gr32+Gr33) / 4, (Gr23+Gr24+Gr33+Gr34) / 4, (Gr31+Gr32+Gr41+Gr42) / 4, (Gr32+Gr33+Gr42+Gr43) / 4, (Gr33+Gr34+Gr43+Gr44) / 4.

[0073] Similarly, the first resolution may be a resolution corresponding to a Nona pattern and the second resolution may be a resolution corresponding to a Quad Bayer pattern. Nine gain table values ​​of the same color can be changed to four target table values.

[0074] FIG. 11 is a diagram for explaining pixels that are excluded when generating a gain table according to an embodiment of the present invention.

[0075] 11, the pixels included in the divided region 1110 may include pixels 1120 that do not maintain linear characteristics with their neighboring pixels. The pixels 1120 that do not maintain linear characteristics with their neighboring pixels may be photodiode (PD) pixels for phase detection autofocus (PDAF) and their neighboring pixels.

[0076] The positions of photodiode pixels for phase detection autofocus can be predetermined in the image sensor. Photodiode pixels for phase detection autofocus have pixel characteristics that change depending on the position, so linear characteristics may not be maintained with adjacent pixels. Photodiode pixels for phase detection autofocus can have two pixels corresponding to one lens (2x1 on chip lens) or four pixels corresponding to one lens (2x2 on chip lens). On the other hand, normal pixels can have one pixel corresponding to one lens (1x1 on chip lens).

[0077] 7, the gain table generator 210 may generate a gain table for pixels included in the region of interest, excluding photodiode pixels for phase detection autofocus. The pixel characteristics of the pixels for phase detection autofocus change depending on the position in the image, and therefore may be irrelevant to the execution of a calibration operation for eliminating color variations.

[0078] FIG. 12 is a flowchart illustrating a noise calibration method according to an embodiment of the present invention.

[0079] According to an embodiment of the present invention, an image sensing device may perform a calibration operation to remove noise contained in a captured image by binning a gain table corresponding to a maximum resolution in low light or video mode.

[0080] In operation S1201, the gain table generator 210 may generate a gain table including gain table values ​​corresponding to a first resolution using pixel values ​​included in a region of interest included in an image. The region of interest may be an area of ​​a predetermined size around any one of intersections of a plurality of grid lines superimposed on the image. The gain table generator 210 may set a plurality of grid lines superimposed on the input image. The gain table generator 210 may set a predetermined region of interest at the intersection of the grid lines.

[0081] The gain table may include gain table values ​​corresponding to a plurality of gain regions. The gain table generator 210 may calculate gain table values ​​by dividing the average value of pixel values ​​of the same color as pixels corresponding to each gain region among pixel values ​​included in the region of interest by the average value of pixel values ​​corresponding to the gain region among pixel values ​​included in the region of interest. The gain table generator 210 may generate a gain table including the gain table values.

[0082] The method of generating the gain table can correspond to the explanations of FIGS.

[0083] In operation S1203, the gain table binning unit 220 may convert the gain table into a target table including target table values ​​corresponding to the second resolution. The target table may include a plurality of calculation regions. The gain table binning unit 220 may determine gain table values ​​corresponding to the plurality of calculation regions. The gain table binning unit 220 may calculate a target table value that is an average value of the gain table values ​​corresponding to the plurality of calculation regions among the gain table values. In another embodiment of the present invention, the gain table binning unit 220 may calculate a median value that is located in the middle of the gain table values ​​corresponding to the plurality of calculation regions among the gain table values. The gain table binning unit 220 may generate a target table including the calculated target table values.

[0084] The method for generating the target table can correspond to the explanations of FIGS.

[0085] In operation S1205, the calibration unit 230 may perform a calibration operation to remove noise included in the image based on the target table. The calibration unit 230 may apply target table values ​​to a plurality of pixel values ​​included in the image. As a result of the calibration operation by the calibration unit 230, color deviation between pixels of the same color may be reduced, thereby removing noise.

[0086] FIG. 13 is a flowchart illustrating a method for binning a gain table according to an embodiment of the present invention.

[0087] The gain table binning method according to an embodiment of the present invention can be performed by a gain table binning unit.

[0088] In operation S1301, the gain table binning unit 220 may determine gain table values ​​corresponding to a plurality of calculation regions. The number of gain table values ​​corresponding to the plurality of calculation regions may be determined based on a first resolution and a second resolution. For example, the first resolution may be a resolution corresponding to a hexadeca pattern, and the second resolution may be a resolution corresponding to a quad-Bayer pattern. The gain table binning unit 220 may determine four gain table values ​​corresponding to one of the plurality of calculation regions based on the first resolution and the second resolution. The four gain table values ​​may be of the same color. The four gain table values ​​may be adjacent gain table values.

[0089] In operation S1303, the gain table binning unit 220 may calculate target table values ​​based on gain table values ​​corresponding to a plurality of calculation regions. The target table may include a plurality of calculation regions. The gain table binning unit 220 may calculate a target table value that is an average value of the gain table values ​​corresponding to a plurality of calculation regions among the gain table values. In another embodiment of the present invention, the gain table binning unit 220 may calculate a median value that is located in the middle of the gain table values ​​corresponding to a plurality of calculation regions among the gain table values. The target table value corresponding to a calculation region may be the average value or median value of the gain table values ​​corresponding to the calculation region. The gain table binning unit 220 may generate a target table including the calculated target table values.

[0090] The binning method of the gain table can correspond to the explanations of FIGS.

[0091] According to an embodiment of the present invention, a gain table can be generated for use in a calibration operation to reduce maze noise at the maximum resolution of an input image. The maze noise may be a grid-like noise caused by color variations between pixels of the same color among the plurality of pixels. Typically, the grid noise is caused by pixel crosstalk. The causes of the crosstalk include electron diffusion in the photodiode and the readout circuit.

[0092] Since variations between pixels appear as a regular pattern, color deviation can be eliminated by applying a gain value to each pixel. A target table can be generated by binning the generated gain table. Since multiple gain tables are not saved, image processing speed is improved and storage space is saved. Since gain tables corresponding to other resolutions are not generated, the time required for calibration can be reduced.

[0093] According to an embodiment of the present invention, in a multi-pattern (Quad, Nona, Hexa-deca), it is possible to minimize the deviation between pixels of the same color, and in a Bayer pattern, it is possible to minimize the color deviation between green Gr, Gb pixels.

[0094] FIG. 14 is a block diagram illustrating a computing system including an image sensor according to an embodiment of the present invention.

[0095] 14, a computing system 2000 includes an image sensor 2010, a processor 2020, a storage device 2030, a memory device 2040, an input / output device 2050, and a display device 2060. Although not shown in FIG. 14, the computing system 2000 may further include a port for communicating with a video card, a sound card, a memory card, a USB device, or the like, or for communicating with other electronic devices.

[0096] The image sensor 2010 can generate image data corresponding to incident light. The display device 2060 can display the image data. The memory device 2030 can store the image data. The processor 2020 can control the operations of the image sensor 2010, the display device 2060, and the memory device 2030.

[0097] The processor 2020 may perform specific calculations or tasks. According to an embodiment of the present invention, the processor 2020 may be a microprocessor or a central processing unit (CPU). The processor 2020 may be connected to and communicate with the storage device 2030, the memory device 2040, and the input / output device 2050 via an address bus, a control bus, and a data bus. According to an embodiment of the present invention, the processor 2020 may be connected to an expansion bus such as a Peripheral Component Interconnect (PCI) bus.

[0098] The storage device 2030 may include a flash memory device, a solid state drive (SSD), a hard disk drive (HDD), a CD-ROM, and any form of non-volatile memory device.

[0099] The memory device 2040 may store data necessary for the operation of the computing system 2000. For example, the memory device 2040 may include volatile memory devices such as dynamic random access memory (DRAM) and static random access memory (SRAM), and non-volatile memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices.

[0100] The input / output device 2050 may include input means such as a keyboard, keypad, or mouse, and output means such as a printer or display.

[0101] The image sensor 2010 may be connected to and communicate with the processor 2020 via the bus or another communication link.

[0102] The image sensor 2010 performs binning on a plurality of pixel data generated from a plurality of pixels included in a pixel array, thereby generating binned pixel data that is evenly distributed in the pixel array.

[0103] The image sensor 2010 may be implemented in various types of packages. For example, at least a portion of the configuration of the image sensor 2010 may be embodied using a package such as a PoP (Package on Package), Ball Grid Arrays (BGAs), Chip Scale Packages (CSPs), Plastic Leaded Chip Carrier (PLCC), Plastic Dual In-Line Package (PDIP), Die in Waffle Pack, Die in Wafer Form, Chip On Board (COB), Ceramic Dual In-Line Package (CERDIP), Plastic Metric Quad Flat Pack (MQFP), Thin Quad Flat Pack (TQFP), Small Outline Integrated Circuit (SOIC), Shrink Small Outline Package (SSOP), Thin Small Outline Package (TSOP), Thin Quad Flat Pack (TQFP), System In Package (SIP), Multi Chip Package (MCP), Wafer-level Fabricated Package (WFP), or Wafer-Level Processed Stack Package (WSP).

[0104] Depending on the embodiment, the image sensor 2010 may be integrated with the processor 2020 on one chip, or may be integrated on separate chips.

[0105] On the other hand, the computing system 2000 should be construed as any computing system that utilizes the image sensor 2010. For example, the computing system 2000 may include a digital camera, a mobile phone, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a smartphone, etc. [Explanation of symbols]

[0106] 10 Image sensing device 100 image sensors 200 Image Processor

Claims

1. a gain table generator that generates a gain table including gain table values ​​corresponding to a first resolution using pixel values ​​included in a region of interest included in an image; a gain table binning unit for converting the gain table into a target table including target table values ​​corresponding to a second resolution; a calibration execution unit that performs a calibration operation to remove noise included in the image based on the target table, 10. An image processing device, wherein the noise is lattice noise caused by crosstalk between a plurality of pixels included in an image sensor.

2. The gain table is including a plurality of gain regions; The gain table values ​​are The image processing device according to claim 1 , wherein the values ​​correspond to the plurality of gain regions, respectively.

3. The gain table values ​​for each of the gain regions are:

3. The image processing device of claim 2, wherein the gain value is calculated by dividing an average value of pixel values ​​of the same color as the pixel values ​​corresponding to the gain area by an average value of pixel values ​​included in the region of interest that correspond to the gain area.

4. The gain table generation unit 4. The image processing apparatus according to claim 3, wherein a gain map for the entire image is generated based on the gain table.

5. The gain table generation unit The image processing apparatus of claim 1 , wherein the gain table is generated based on pixel values ​​other than pixel values ​​corresponding to pixels for phase detection autofocus within the region of interest.

6. The image processing apparatus of claim 1 , further comprising a gain table storage unit for storing the gain table.

7. The first resolution is:

2. The image processing apparatus according to claim 1, wherein the resolution is higher than the second resolution.

8. The number of gain table values ​​is 2. The image processing apparatus according to claim 1, wherein the number of target table values ​​is a positive integer multiple of the number of target table values.

9. The first resolution is: With a resolution corresponding to a hexadeca pattern, The second resolution is 2. The image processing device according to claim 1, wherein the resolution is one of a resolution corresponding to a nonacell pattern, a resolution corresponding to a quad-Bayer pattern, and a resolution corresponding to a Bayer pattern.

10. The first resolution is: With a resolution that corresponds to the Nonacelle pattern, The second resolution is 2. The image processing device according to claim 1, wherein the resolution corresponds to a quad-Bayer pattern or a Bayer pattern.

11. The target table is It includes multiple computational domains, The target table values ​​in each of the computational domains are:

2. The image processing apparatus according to claim 1, wherein the gain table value is an average value of the gain table values ​​corresponding to the calculation region.

12. The number of gain table values ​​corresponding to the plurality of calculation domains is The image processing device of claim 11 , wherein the first resolution and the second resolution are determined based on the first resolution and the second resolution.

13. The target table is It includes multiple computational domains, The target table values ​​in each of the computational domains are:

2. The image processing apparatus according to claim 1, wherein the calculation is performed based on a median value of the gain table values ​​corresponding to the calculation region.

14. The number of gain table values ​​corresponding to the calculation domain is The image processing device of claim 13 , wherein the first resolution and the second resolution are determined based on the first resolution and the second resolution.

15. The calibration execution unit 2. The image processing apparatus of claim 1, wherein the target table values ​​are applied to a plurality of pixel values ​​contained in the image.

16. The calibration execution unit 2. The image processing apparatus of claim 1, wherein a plurality of pixel values ​​included in the image are multiplied by the target table values ​​corresponding to the second resolution, and the target table values ​​are generated by binning the gain table values ​​corresponding to the first resolution.

17. The region of interest is 2. The image processing apparatus according to claim 1, wherein the area is a region of a predetermined size around one of the intersections of a plurality of grid lines superimposed on the image.

18. an image sensor that acquires an image using a plurality of pixels; an image processor that generates a gain table including gain table values ​​corresponding to a first resolution using pixel values ​​included in a region of interest included in the image, converts the gain table into a target table including target table values ​​corresponding to a second resolution lower than the first resolution, and removes noise included in the image using the target table; The image sensing device, wherein the noise is lattice noise caused by crosstalk between the plurality of pixels.

19. The target table value is 20. The image sensing device of claim 18, wherein the gain table value is generated using an average or median value of at least two or more gain table values.

Images