Image processing device and image sensing device

Abstract

The present disclosure relates to an image processing device and an image sensing device. An image sensing device and an operation method thereof may be provided herein. An image sensing device may include: an image sensor that uses a plurality of pixels to obtain an image; and an image processor that is configured to generate a gain table including gain table values ​​corresponding to a first resolution using pixel values ​​included in a region of interest included in the image, convert the gain table into a target table including target table values ​​corresponding to a second resolution lower than the first resolution, and eliminate noise included in the image based on the target table.

Description

Technical Field

[0001] Various embodiments of the present disclosure relate generally to electronic devices, and more particularly, to an image sensing device and a method of operating the image sensing device. Background Art

[0002] Image sensors are devices that capture images using the properties of semiconductors that react to light. With the recent development of the computer industry and the communications industry, there is a growing demand for image sensors with improved performance in various fields such as smartphones, digital cameras, gaming devices, the Internet of Things, robots, security cameras, and medical miniature cameras.

[0003] Image sensors can be roughly classified into charge coupled device (CCD) image sensors and complementary metal oxide semiconductor (CMOS) image sensors. Compared with CMOS image sensors, CCD image sensors have less noise and better image quality. However, CMOS image sensors are simply driven and can be implemented in various scanning methods. In addition, CMOS image sensors make it possible to integrate their signal processing circuits in a single chip, so that the product is easy to miniaturize and has low power consumption, and has low manufacturing costs due to the ability to use CMOS process technology compatibly. Due to its properties suitable for mobile devices, CMOS image sensing devices are now widely used. Summary of the invention

[0004] Various embodiments of the present disclosure are directed to an image sensing device and a method of operating the image sensing device, which are performed with improved grid noise removal.

[0005] An embodiment of the present disclosure may provide an image processing device. The image processing device may include: a gain table generator configured to generate a gain table including gain table values ​​corresponding to a first resolution using pixel values ​​included in a region of interest included in an image; a gain table binner configured to convert the gain table into a target table including target table values ​​corresponding to a second resolution; and a calibrator configured to perform a calibration operation to eliminate noise included in the image based on the target table.

[0006] An embodiment of the present disclosure may provide an image sensing device. The image sensing device may include: an image sensor that uses a plurality of pixels to obtain an image; and an image processor that is configured to generate a gain table including gain table values ​​corresponding to a first resolution using pixel values ​​included in a region of interest included in the image, convert the gain table into a target table including target table values ​​corresponding to a second resolution lower than the first resolution, and eliminate noise included in the image based on the target table.

[0007] Embodiments of the present disclosure may provide an operating method for an image processing device. The operating method may include the following steps: generating an M×M gain value array corresponding to a region of interest (ROI) of a 2M×2M pixel value array within an image; generating an N×N target value array; and eliminating noise from the image. The value of M is K times greater than the value of N. The gain value within the gain value array is obtained by dividing the first statistical representative value by the second statistical representative value. The first statistical representative value has a pixel value having the same color as a reference pixel value within the ROI corresponding to each gain value. The second statistical representative value has a reference pixel value. The target value within the target value array is a third statistical representative value of the gain value of a K×K sub-gain value array corresponding to the target value, and the K×K sub-gain value array belongs to the M×M gain value array. Noise is eliminated by multiplying the target value with the pixel value of a K×K sub-image array corresponding to each target value, and the K×K sub-image array belongs to the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a diagram illustrating an image sensing device according to an embodiment of the present disclosure.

[0009] Figure 2 The present invention is shown in the embodiment of the present invention. Figure 1 Figure 1. Image sensor diagram of an image sensor.

[0010] Figure 3 The present invention is shown in the embodiment of the present invention. Figure 2 Figure 2 is a diagram of a Bayer pattern of a pixel array.

[0011] Figure 4 The present invention is shown in the embodiment of the present invention. Figure 2 Figure 2. A diagram of a quad Bayer pattern for a pixel array.

[0012] Figure 5 The present invention is shown in the embodiment of the present invention. Figure 2 Figure 2 shows a nine-cell pattern of a pixel array.

[0013] Figure 6 The present invention is shown in the embodiment of the present invention. Figure 2 A diagram of a sixteen-element pattern of a pixel array.

[0014] Figure 7 is a block diagram of an image sensing device according to an embodiment of the present disclosure.

[0015] Figure 8 is a diagram showing a region of interest set according to an embodiment of the present disclosure.

[0016] Fig. 9 is a diagram illustrating a method of generating a gain table according to an embodiment of the present disclosure.

[0017] Fig.10 is a diagram illustrating a method of binning a gain table according to an embodiment of the present disclosure.

[0018] Fig.11 is a diagram illustrating pixels excluded from generation of a gain table according to an embodiment of the present disclosure.

[0019] Fig.12 is a flow chart illustrating a noise calibration method according to an embodiment of the present disclosure.

[0020] Fig.13 is a flow chart illustrating a gain table binning method according to an embodiment of the present disclosure.

[0021] Fig.14 is a block diagram of a computing system including an image sensor according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The specific structural or functional descriptions in the embodiments of the present disclosure introduced in this specification are only for describing the embodiments of the present disclosure, and these descriptions should not be interpreted as being limited to the embodiments described in this specification.

[0023] Embodiments of the present disclosure will now be described with reference to the accompanying drawings to describe the technical concept of the present disclosure so that those skilled in the art can easily put it into practice.

[0024] Figure 1 is a diagram illustrating an image sensing device according to an embodiment of the present disclosure.

[0025] Reference Figure 1 , the image sensing device 10 may include an image sensor 100 and an image processor 200 .

[0026] The image sensing device 10 may be included in an electronic device such as a digital camera, a mobile phone, a smart phone, a tablet personal computer (tablet PC), a notebook, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital video camera, a portable multimedia player (PMP), a mobile Internet device (MID), a PC, a wearable device, or a camera for various purposes (front-mounted vehicle camera, rear-mounted vehicle camera, black box camera, etc.).

[0027] The image sensor 100 may be implemented using a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. The image sensor 100 may generate image data of an object (not shown) input (or captured) through a lens (not shown). The lens may include at least one lens constituting an optical system.

[0028] The image sensor 100 may include a plurality of pixels. The image sensor 100 may generate a plurality of pixel values ​​DPXs corresponding to a captured image from the plurality of pixels. The plurality of pixel values ​​DPXs generated by the image sensor 100 may be sent to the image processor 200. For example, the image sensor 100 may generate a plurality of pixel values ​​DPXs corresponding to a single frame.

[0029] The image processor 200 may control the image sensor 100. Specifically, the image processor 200 may perform processing to enhance the image quality of pixel data received from the image sensor 100 and output the processed image data. The processing may include electronic image stabilization (EIS), interpolation, tone correction, image quality correction, size adjustment, etc. The image processor 200 may also be referred to as an image processing device.

[0030] The image processor 200 may calibrate the noise included in the image based on the system information AG and the plurality of pixel values ​​DPXs. The image may include maze noise. Maze noise may be a grid-type noise caused by color imbalance between pixels of the same color among a plurality of pixels. Maze noise may be caused by color difference between pixels having the same color. In an embodiment of the present disclosure, maze noise may be a grid noise from crosstalk between a plurality of pixels.

[0031] The image processor 200 may calculate a calibration gain value for performing a calibration operation to reduce the maze noise based on the plurality of pixel values.

[0032] exist Figure 1 In the embodiment of the present disclosure, the image processor 200 may be implemented in a chip separate from the image sensor 100. In this case, the chip of the image sensor 100 and the chip of the image processor 200 may be implemented in a package (e.g., a multi-chip package). In an embodiment of the present disclosure, the image processor 200 may be included as a part of the image sensor 100 so that the image processor 200 and the image sensor 100 may be implemented in a single chip.

[0033] Figure 2 The present invention is shown in the embodiment of the present invention. Figure 1 Figure 1. Image sensor diagram of an image sensor.

[0034] Reference Figure 2 , the image sensor 100 may include a pixel array 110 , a row decoder 120 , a timing generator 130 , and a signal transducer 140 .

[0035] The pixel array 110 may include a color filter array 111 and a photoelectric conversion layer 113 formed below the color filter array 111 and including a plurality of photoelectric conversion elements corresponding to respective pixels of the color filter array 111. The pixel array 110 may include a plurality of pixels that output color information included in incident light. Each of the plurality of pixels may output a pixel signal corresponding to the incident light that has passed through the corresponding color filter array 111.

[0036] The color filter array 111 may include a color filter that passes only light of a specific wavelength (e.g., red, blue, and green) incident on each pixel. Using the color filter array 111, the pixel data of each pixel may represent a value corresponding to the intensity of the specific wavelength of light, and each pixel may be referred to as a red pixel R, a blue pixel B, or a green pixel G according to the type of the color filter.

[0037] Specifically, each of the plurality of pixels may accumulate photocharges generated according to incident light and generate a pixel signal corresponding to the accumulated photocharges. Each pixel may include a photoelectric conversion element (e.g., a photodiode, a phototransistor, a photogate, or a pinned photodiode) for converting a photosignal into an electrical signal and at least one transistor for processing the electrical signal.

[0038] The pixel array 110 may include a plurality of pixels arranged in rows and columns. The pixel array 110 may generate a plurality of pixel signals VPX for each row. The plurality of pixel signals VPX may each be an analog pixel signal VPX.

[0039] The row decoder 120 may select one of a plurality of rows of a plurality of pixel arrangements in response to an address and a control signal output from the timing generator 130 .

[0040] The signal transducer 140 may convert a plurality of analog pixel signals VPX into a plurality of digital pixel values ​​DPXs. The plurality of digital pixel values ​​DPXs may be output in various patterns. In response to a control signal output from the timing generator 130, the signal transducer 140 may perform correlated double sampling (CDS) on each signal output from the pixel array 110, perform analog-to-digital conversion on the signal passed through the CDS, and output a digital signal. Each digital signal may be a signal corresponding to the intensity of incident light of a wavelength that has passed through the corresponding color filter array 111.

[0041] The signal transducer 140 may include a CDS block and an analog-to-digital converter (ADC) block. The CDS block may instantaneously sample and hold a set of reference signals and image signals provided for each of a plurality of column lines included in the pixel array 110. In other words, the CDS block may sample and hold the levels of the image signal and the reference signal corresponding to each column. The ADC block may output pixel data obtained by converting the CDS signal of each column output from the CDS block into a digital signal. To this end, the ADC block may include a comparator and a counter corresponding to each column.

[0042] In addition, in an embodiment of the present disclosure, the image sensor 100 may further include an output buffer 150. The output buffer 150 may be implemented using many buffers for storing digital signals output from the signal transducer 140. Specifically, the output buffer 150 may latch and output pixel data in units of columns provided from the signal transducer 140. The output buffer 150 may temporarily store the pixel data output from the signal transducer 140, and sequentially output the pixel data under the control of the timing generator 130. In some embodiments of the present disclosure, the output buffer 270 may be omitted.

[0043] Figure 3 The present invention is shown in the embodiment of the present invention. Figure 2 Figure 1. A diagram of the Bayer pattern of a pixel array.

[0044] Reference Figure 3 , the pixel array 110 may be arranged in a predetermined pattern. For example, the pixel array 110 may be arranged in a Bayer pattern. Figure 3 As indicated by dotted lines in FIG. 1 , the Bayer pattern may be a repetition of a unit of 2×2 pixels. In each unit, two pixels Gb and Gr with a green filter may be arranged diagonally, and a pixel B with a blue filter and a pixel R with a red filter may be arranged in the remaining corner. The four pixels B, Gb, Gr, and R are not limited to Figure 3 The arrangement shown here may instead be arranged differently based on a Bayer pattern.

[0045] Figure 4 The present invention is shown in the embodiment of the present invention. Figure 2 Figure 2. A diagram of a quad Bayer pattern for a pixel array.

[0046] Reference Figure 4 , the pixel array 110 may be arranged in a predetermined pattern. For example, the pixel array 110 may be arranged in a quaternary Bayer pattern. Figure 4 As indicated by dashed lines in FIG. 1 , the quaternary Bayer pattern may be a repetition of a unit of 4×4 pixels. In each unit, eight pixels Gb and Gr with a green color filter may be arranged diagonally, and four pixels B with a blue color filter and four pixels R with a red color filter may be arranged in the remaining corners. The sixteen pixels B, Gb, Gr, and R are not limited to Figure 4 The arrangement shown, but can be arranged differently based on a quaternary Bayer pattern.

[0047] Figure 5 The present invention is shown in the embodiment of the present invention. Figure 2 Figure 2 shows a nine-cell pattern of a pixel array.

[0048] Reference Figure 5 , the pixel array 110 may be arranged in a predetermined pattern. For example, the pixel array 110 may be arranged in a nine-unit pattern. Figure 5 As indicated by dashed lines in FIG. 1 , the nine-unit pattern may be a repetition of a unit of 6×6 pixels. In each unit, eighteen pixels Gb and Gr with a green color filter may be arranged diagonally, and nine pixels B with a blue color filter and nine pixels R with a red color filter may be arranged in the remaining corners. The thirty-six pixels B, Gb, Gr, and R are not limited to Figure 5 The arrangement shown, but can be arranged differently based on the nine-cell pattern.

[0049] Figure 6 The present invention is shown in the embodiment of the present invention. Figure 2 A diagram of a sixteen-element pattern of a pixel array.

[0050] Reference Figure 6 , the pixel array 110 may be arranged in a predetermined pattern. For example, the pixel array 110 may be arranged in a hexadecimal pattern. Figure 6 As indicated by dashed lines in FIG. 1 , the sixteen-element pattern may be a repetition of a unit of 8×8 pixels. In each unit, thirty-two pixels Gb and Gr with a green color filter may be arranged diagonally, and sixteen pixels B with a blue color filter and sixteen pixels R with a red color filter may be arranged in the remaining corners. The sixty-four pixels B, Gb, Gr, and R are not limited to Figure 6 The arrangement shown, but can be arranged differently based on the hexadecimal pattern.

[0051] Figure 7 is a block diagram of an image sensing device according to an embodiment of the present disclosure.

[0052] Reference Figure 7 , the image sensing device may include an image sensor 100 and an image processor 200. The image sensor 100 for obtaining an image using a plurality of pixels may send a plurality of pixel values ​​to the image processor 200. The image processor 200 may remove noise included in the image based on the plurality of pixel values. The image processor 200 may include a gain table generator 210, a gain table binner 220, a calibrator 230, and a gain table storage unit 240. The image sensor 100 and the image processor 200 may correspond to a combination of Figure 1 and Figure 2 The content of the description.

[0053] The gain table generator 210 may generate a gain table including gain table values ​​corresponding to the first resolution using pixel values ​​included in the region of interest included in the image. In an embodiment of the present disclosure, the gain table generator 210 may generate a gain table based on a plurality of pixel values, the gain table being used for a calibration operation to reduce noise occurring in the full resolution of the image. The noise may include maze noise. Maze noise may be a grid-type noise caused by a difference in pixel values ​​between pixels having the same color among a plurality of pixels. Pixel crosstalk may cause a difference in pixel values ​​between pixels having the same color.

[0054] The region of interest may be a region of predetermined size around one of the intersections of a plurality of grid lines superimposed on the image. The gain table generator 210 may set a plurality of grid lines to be superimposed on the input image. The gain table generator 210 may set a predetermined region of interest around the intersection of the grid lines. The gain table generator 210 may generate a gain table corresponding to the full resolution in the region of interest. The gain table generator 210 may generate a gain map of the entire image based on the gain table. Specifically, the gain table generator 210 may use a linear interpolation method to expand the gain table into a gain map.

[0055] The gain table may be stored in the gain table storage unit 240. The gain table generator 210 may generate the gain table by excluding pixel values ​​corresponding to pixels for phase detection auto focusing (PDAF) from pixel values ​​included in the region of interest, which will be referred to as Fig.11 discuss.

[0056] The gain table may include a plurality of gain regions. The gain table values ​​may each be a value corresponding to each of the plurality of gain regions. The gain table values ​​of the respective gain regions may each be obtained by dividing the average value of pixel values ​​having the same color as the pixel value corresponding to the gain region among the pixel values ​​included in the region of interest by the average value of the pixel values ​​corresponding to the gain region. The gain table generator 210 may generate a gain table including a plurality of gain table values.

[0057] The gain table binner 220 may convert the gain table into a target table including target table values ​​corresponding to the second resolution. In an embodiment of the present disclosure, the first resolution is higher than the second resolution. The first resolution may be a full resolution, and the second resolution may be a target resolution. For example, the first resolution may be a resolution corresponding to a sixteen-element pattern, and the second resolution may be a resolution corresponding to a nine-element pattern, a resolution corresponding to a four-element Bayer pattern, and one of a resolution corresponding to a Bayer pattern. In an example, the first resolution may be a resolution corresponding to a nine-element pattern, and the second resolution may be a resolution corresponding to a four-element Bayer pattern or a resolution corresponding to a Bayer pattern. The full resolution may be used for high-intensity lighting with sufficient light, and the binned Bayer pattern may be used for low-intensity lighting with insufficient light or for capturing in video mode.

[0058] The target table may include a plurality of calculation areas. Each target table value for each calculation area may be an average value of the gain table values ​​corresponding to each calculation area. The number of gain table values ​​corresponding to the plurality of calculation areas may be determined based on the first resolution and the second resolution. For example, when the first resolution is a resolution corresponding to a sixteen-element pattern and the second resolution is a resolution corresponding to a four-element Bayer pattern, the number of table values ​​corresponding to the plurality of calculation areas may be four.

[0059] In an embodiment of the present disclosure, each target table value of each calculation region is a median value of the gain table values ​​corresponding to the calculation region.The number of gain table values ​​corresponding to each of the plurality of calculation regions may be determined based on the first resolution and the second resolution.

[0060] The gain table binner 220 may calculate a target table value, which is an average of those gain table values ​​corresponding to a plurality of gain regions. In an embodiment of the present disclosure, the gain table binner 220 may calculate a median of those gain table values ​​corresponding to a plurality of calculation regions. The target table value corresponding to a calculation region may be an average or median of the gain table values ​​corresponding to the calculation region. The gain table binner 220 may generate a target table including the calculated target table values. In an embodiment of the present disclosure, the number of gain table values ​​may be a positive integer multiple of the number of target table values.

[0061] In an embodiment of the present disclosure, the gain table binner 220 may determine gain table values ​​corresponding to a plurality of calculation regions. For example, the first resolution may be a resolution corresponding to a sixteen-element pattern, and the second resolution may be a resolution corresponding to a four-element Bayer pattern. The gain table binner 220 may determine four gain table values ​​corresponding to one of the plurality of calculation regions based on the first resolution and the second resolution. The four gain table values ​​may have the same color. The four gain table values ​​may be adjacent gain table values.

[0062] The calibrator 230 may perform a calibration operation based on the target table to eliminate noise included in the image. The calibrator 230 may apply the target table value to a plurality of pixel values ​​included in the image. For example, the calibrator 230 may multiply the plurality of pixel values ​​included in the image by the target table value in units of the second resolution.

[0063] In an embodiment of the present disclosure, the calibrator 230 may perform a calibration operation based on the target table to reduce the maze noise appearing in the second resolution of the input image. As a result of the calibration operation of the calibrator 230, the color difference between pixels having the same color may be reduced. Therefore, the grid noise may be eliminated.

[0064] In an embodiment of the present disclosure, the image sensing device 10 may generate a gain table including a gain table value corresponding to a first resolution using a pixel value included in a region of interest included in an image. The image sensing device 10 may convert the gain table into a target table including a target table value corresponding to a second resolution (lower than the first resolution). The image sensing device 10 may eliminate noise included in the image based on the target table. The target table value may be generated using an average or median value of at least two gain table values.

[0065] Figure 8 is a diagram showing a region of interest set according to an embodiment of the present disclosure.

[0066] Reference Figure 8 , a gain table is generated for use in performing a calibration operation on a portion of an input image so that a gain map for the entire image can be derived.

[0067] When calculating the calibration gain value, calculating the gain values ​​of all pixels included in the image may slow down the image processing rate and require a large storage space. In order to increase the image processing rate and save storage space, a grid method can be used.

[0068] A plurality of grid lines 810 may be set in the input image 800. The plurality of grid lines 810 may be superimposed on the image 800. Intersections 820 of the grid lines 810 may be set on the image 800. A predetermined region of interest 830 may be set for the intersections 820. The size of the region of interest 830 may vary according to the image.

[0069] exist Figure 8 In the example, there may be multiple intersection points 820, so the location of the region of interest 830 may vary. Once the gain table is generated for the region of interest 830, the gain table may be expanded into a gain map for the entire image 800 according to a scheme such as linear interpolation. Figure 8 The region of interest can be set by the gain table generator.

[0070] Fig. 9is a diagram illustrating a method of generating a gain table according to an embodiment of the present disclosure.

[0071] Fig. 9 An example is shown in which the first resolution of the input image is a resolution corresponding to a hexadecimal pattern and the size of the region of interest 830 is 16×16 pixels.

[0072] The gain table generator may divide the region of interest 830 to correspond to the first resolution. Since the full resolution of the image corresponds to a sixteen-element pattern, the region of interest 830 may be divided into four sub-regions. Fig. 9 , reference numeral 910 indicates one of the four sub-regions within the region of interest 830 .

[0073] The gain table 920 may include gain table values ​​corresponding to the plurality of gain regions, respectively. For example, the gain table value may correspond to the gain region 921 among the plurality of gain regions. The gain table value corresponding to the gain region 921 may be a value obtained by dividing the average value of pixel values ​​having the same color as the pixel value corresponding to the gain region 921 (i.e., any one of 911, 913, 915, and 917) among the pixel values ​​included in the region of interest by the average value of the pixel values ​​911, 913, 915, and 917 corresponding to the gain region 921. Fig. 9 In the example, when the pixel value corresponding to the gain region 921 (i.e., any one of 911, 913, 915, and 917) has a green Gr, the average value of the pixel values ​​of the green Gr included in the region of interest may be calculated. In an embodiment of the present disclosure, the gain table value corresponding to the gain region 921 may be a value obtained by dividing the average value of the pixel values ​​having the same color as the pixel value corresponding to the gain region 921 (i.e., any one of 911, 913, 915, and 917) among the pixel values ​​included in the region of interest by the median value of the pixel values ​​911, 913, 915, and 917 corresponding to the gain region 921.

[0074] Likewise, the remaining gain table values ​​of the gain table 920 may be calculated. In an embodiment of the present disclosure, the color of the pixel included in the image sensor may be any one of green Gr and Gb, red R, and blue B. When generating the gain table, the green Gr and Gb pixels have the same color. The inconsistency between the green Gr and Gb pixels may cause noise. Fig. 9 The operation of generating the gain table can be performed by a gain table generator.

[0075] Fig.10 is a diagram illustrating a method of binning a gain table according to an embodiment of the present disclosure.

[0076] Fig.10An example is shown in which the first resolution is a resolution corresponding to a hexadecimal pattern and the second resolution is a resolution corresponding to a quaternary Bayer pattern or a Bayer pattern. The gain table 920 corresponding to the first resolution may be subjected to binning to become target tables 1010 and 1020 . Fig.10 The binning method corresponds to combining Figure 7 The content of the description.

[0077] Reference Fig.10 , the gain table binner 220 may convert the gain table into a target table including target table values ​​corresponding to the second resolution. The target table may include a plurality of calculation regions. The gain table values ​​may correspond to the plurality of calculation regions. The number of gain table values ​​corresponding to the plurality of calculation regions may be determined based on the first resolution and the second resolution. The gain table binner 220 may determine the gain table values ​​corresponding to the plurality of calculation regions.

[0078] The gain table binner 220 may calculate a target table value, which is an average value of those gain table values ​​corresponding to a plurality of gain regions. In an embodiment of the present disclosure, the gain table binner 220 may calculate a median value at the center of those gain table values ​​corresponding to a plurality of calculation regions. The target table value corresponding to a calculation region may be an average value or a median value of the gain table values ​​corresponding to the calculation region. The gain table binner 220 may generate a target table including the calculated target table values.

[0079] Reference Fig.10 , the gain table 920 may be converted into a target table 1010. The second resolution may be a resolution corresponding to a quaternary Bayer pattern. The gain table binner 220 may determine four gain table values ​​corresponding to one of the plurality of calculation regions based on the first resolution and the second resolution. The four gain table values ​​may have the same color. The four gain table values ​​may be adjacent gain table values.

[0080] For example, four gain table values ​​of the same color may be binned. The table value Gr1 in the target table is represented as follows: Gr1 = (Gr11 + Gr12 + Gr21 + Gr22) / 4. Similarly, the table value Gr2 in the target table is represented as follows: Gr2 = (Gr13 + Gr14 + Gr23 + Gr24) / 4. Similarly, the remaining table values ​​Gr3, Gr4, R1, R2, R3, R4, B1, B2, B3, B4, Gb1, Gb2, Gb3, and Gb4 of the target table 1010 may be calculated.

[0081] Reference Fig.10 , the gain table 920 may be converted into a target table 1020. The second resolution may be a resolution corresponding to a Bayer pattern. The gain table binner 220 may determine sixteen gain table values ​​corresponding to one of the plurality of calculation regions based on the first resolution and the second resolution. The sixteen gain table values ​​may have the same color. The sixteen gain table values ​​may be adjacent gain table values.

[0082] For example, sixteen gain table values ​​of the same color may be binned. The gain table value Gr in the target table is expressed as follows: Gr=(Gr11+Gr12+Gr13+Gr14+Gr21+Gr22+Gr23+Gr24+Gr31+Gr32+Gr33+Gr34+Gr41+Gr42+Gr43+Gr44) / 16. Similarly, the gain table value Gb in the target table is expressed as follows: Gb=(Gb11+Gb12+Gb13+Gb14+Gb21+Gb22+Gb23+Gb24+Gb31+Gb32+Gb33+Gb34+Gb41+Gb42+Gb43+Gb44) / 16. In the resolution corresponding to the Bayer pattern, inconsistency occurs only between green Gr and Gb, and thus the remaining table values ​​R and B of the target table 1020 may not be calculated.

[0083] In an embodiment of the present disclosure, the gain table binner 220 may perform binning on the gain table corresponding to the first resolution, and thus may not store the gain table corresponding to the second resolution.

[0084] In an embodiment of the present disclosure, a gain table value corresponding to a calculation region may correspond to another calculation region. That is, a single gain table value may correspond to multiple calculation regions. For example, the first resolution may be a resolution corresponding to a sixteen-element pattern, and the second resolution may be a resolution corresponding to a nine-element pattern. In other words, sixteen gain table values ​​of the same color may be binned into nine target table values. The nine target table values ​​are expressed as follows: (Gr11+Gr12+Gr21+Gr22) / 4, (Gr12+Gr13+Gr22+Gr23) / 4, (Gr13+Gr14+Gr23+Gr24) / 4, (Gr21+Gr22+G31+Gr32) / 4, (Gr22+Gr23+Gr32+Gr33) / 4, (Gr23+Gr24+Gr33+Gr34) / 4, (Gr31+Gr32+Gr41+Gr42) / 4, (Gr32+Gr33+Gr42+Gr43) / 4 and (Gr33+Gr34+Gr43+Gr44) / 4.

[0085] Similarly, the first resolution may be a resolution corresponding to a nine-element pattern, and the second resolution may be a resolution corresponding to a four-element Bayer pattern. In this case, the same nine gain table values ​​may be converted into four target table values.

[0086] Fig.11 is a diagram illustrating pixels excluded from generation of a gain table according to an embodiment of the present disclosure.

[0087] Reference Fig.11 , in the sub-region 1110 (eg, Fig. 9 Among the pixels in the sub-region 910 of FIG. 1 , there may be a pixel 1120 for which it is difficult to maintain linearity with adjacent pixels. The pixel 1120 for which it is difficult to maintain linearity with adjacent pixels may be a photodiode (PD) pixel for PDAF and its adjacent pixels.

[0088] The position of the PD pixel for PDAF may be predetermined by the image sensor. Since the PD pixel for PDAF may have a property that changes with position, the PD pixel may not maintain linearity with its adjacent pixels. For the PD pixel for PDAF, two pixels may correspond to one lens, i.e., 2×1 on the chip lens, or four pixels may correspond to one lens, i.e., 2×2 on the chip lens. On the other hand, for a normal pixel, a single pixel may correspond to one lens, i.e., 1×1 on the chip lens.

[0089] Reference Figure 7 , the gain table generator 210 may generate a gain table by excluding PD pixels used for PDAF from pixels included in the region of interest. Since pixels used for PDAF have a pixel property that changes with the position of the image, these pixels may be irrelevant to performing a calibration operation to correct color imbalance.

[0090] Fig.12 is a flow chart illustrating a noise calibration method according to an embodiment of the present disclosure.

[0091] In an embodiment of the present disclosure, the image sensing device may perform a calibration operation to eliminate noise included in a captured image. The image sensing device may perform a calibration operation by binning a gain table corresponding to a full resolution in a low intensity illumination or video mode.

[0092] In operation S1201, the gain table generator 210 may generate a gain table including a gain table value corresponding to a first resolution using a pixel value included in a region of interest included in an image. The region of interest may be a region having a predetermined size around one of intersections of a plurality of grid lines superimposed on the image. The gain table generator 210 may set a plurality of grid lines to be superimposed on the input image. The gain table generator 210 may set a predetermined region of interest around the intersection of the grid lines.

[0093] The gain table may include gain table values ​​corresponding to the plurality of gain regions, respectively. The gain table generator 210 may calculate gain table values ​​for the respective gain regions, each of which is obtained by dividing an average value of pixel values ​​having the same color as the pixel value corresponding to the gain region among the pixel values ​​included in the region of interest by an average value of the pixel values ​​corresponding to the gain region. The gain table generator 210 may generate a gain table including the gain table values.

[0094] The method of generating the gain table may correspond to combining Figure 7 , Figure 8 , Fig. 9 and Fig.11 The content of the description.

[0095] In operation S1203, the gain table binner 220 may convert the gain table into a target table including target table values ​​corresponding to the second resolution. The target table may include multiple calculation areas. The gain table binner 220 may determine the gain table values ​​corresponding to the multiple calculation areas. The gain table binner 220 may calculate a target table value, which is an average value of those gain table values ​​corresponding to the multiple gain areas. In an embodiment of the present disclosure, the gain table binner 220 may calculate a median at the center of those gain table values ​​corresponding to the multiple calculation areas. The gain table binner 220 may generate a target table including the calculated target table values.

[0096] The method of generating the target table can correspond to combining Figure 7 and Fig.10 The content of the description.

[0097] In operation S1205, the calibrator 230 may perform a calibration operation based on the target table to eliminate noise included in the image. The calibrator 230 may apply the target table value to a plurality of pixel values ​​included in the image. As a result of the calibration operation of the calibrator 230, the color difference between pixels having the same color may be reduced. Therefore, the noise may be eliminated.

[0098] Fig.13 is a flow chart illustrating a gain table binning method according to an embodiment of the present disclosure.

[0099] The gain table binning method according to an embodiment of the present disclosure may be performed by a gain table binner.

[0100] In operation S1301, the gain table binner 220 may determine gain table values ​​corresponding to a plurality of calculation regions. The number of gain table values ​​corresponding to the plurality of calculation regions may be determined based on a first resolution and a second resolution. For example, the first resolution may be a resolution corresponding to a sixteen-element pattern, and the second resolution may be a resolution corresponding to a four-element Bayer pattern. The gain table binner 220 may determine four gain table values ​​corresponding to one of the plurality of calculation regions based on the first resolution and the second resolution. The four gain table values ​​may have the same color. The four gain table values ​​may be adjacent gain table values.

[0101] In operation S1303, the gain table binner 220 may calculate a target table value based on the gain table values ​​corresponding to the plurality of calculation regions. The target table may include a plurality of calculation regions. The gain table binner 220 may calculate a target table value that is an average value of those gain table values ​​corresponding to the plurality of gain regions. In an embodiment of the present disclosure, the gain table binner 220 may calculate a median value at the center of those gain table values ​​corresponding to the plurality of calculation regions. The target table value corresponding to the calculation region may be an average value or a median value of the gain table values ​​corresponding to the calculation region. The gain table binner 220 may generate a target table including the calculated target table values.

[0102] The gain table binning method can correspond to combining Figure 7 and Fig.10 The content of the description.

[0103] In an embodiment of the present disclosure, a gain table may be generated for a calibration operation to reduce maze noise in the full resolution of an input image. Maze noise may be a grid-type noise caused by color imbalance between pixels of the same color in a plurality of pixels. Grid noise may typically be caused by pixel crosstalk. Crosstalk may have various causes, including electron diffusion in PD, read circuits, etc.

[0104] The imbalance between pixels may occur in a regular pattern, so the color difference can be eliminated by applying a gain value to each pixel. The target table can be generated by binning the gain table. There is no need to store multiple gain tables, so the image processing rate is increased and storage space can be saved. Gain tables corresponding to other resolutions are not generated, thereby reducing the time required for calibration.

[0105] According to an embodiment of the present disclosure, inconsistency between pixels having the same color may be minimized in a multi-pattern (eg, a four-element Bayer pattern, a nine-element pattern, or a sixteen-element pattern), and color differences between green Gr and Gb pixels may be minimized in a Bayer pattern.

[0106] Fig.14 is a block diagram of a computing system including an image sensor according to an embodiment of the present disclosure.

[0107] Reference Fig.14 , the computing system 2000 may include an image sensor 2010, a processor 2020, a storage device 2030, a memory device 2040, an input / output device 2050, and a display device 2060. Fig.14 Not shown, the computing system 2000 may also include ports for communicating with a video card, a sound card, a memory card, a universal serial bus (USB) device, or other various electronic devices.

[0108] The image sensor 2010 may generate image data corresponding to incident light. The display device 2060 may display the image data. The storage device 2030 may store the image data. The processor 2020 may control operations of the image sensor 2010, the display device 2060, and the storage device 2030.

[0109] The processor 2020 may perform specific calculations or tasks. In an embodiment of the present disclosure, the processor 2020 may be a microprocessor or a central processing unit (CPU). The processor 2020 may be coupled to a storage device 2030, a memory device 2040, and an input / output device 2050 via an address bus, a control bus, and a data bus for communication. In an embodiment of the present disclosure, the processor 2020 may also be coupled to an expansion bus such as a peripheral component interconnect (PCI) bus.

[0110] The storage device 2030 may include a flash memory device, a solid state drive (SSD), a hard disk drive (HDD), a compact disk read only memory (CD-ROM), and any type of non-volatile memory device.

[0111] The memory device 2040 may store data required for the operation of the computing system 2000. For example, the memory device 2040 may include volatile memory devices such as dynamic random access memory (DRAM) and static random access memory (SRAM), and nonvolatile memory devices such as erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), and flash memory devices.

[0112] The input / output device 2050 may include input devices such as a keyboard, a keypad, a mouse, etc. and output devices such as a printer, a display, etc.

[0113] Image sensor 2010 may be coupled to processor 2020 via a bus or other communication link for communication.

[0114] The image sensor 2010 may perform binning on a plurality of pixel data generated from a plurality of pixels included in a pixel array to generate binned pixel data uniformly distributed in the pixel array.

[0115] The image sensor 2010 may be implemented in various types of packages. For example, at least some components of the image sensor 2010 may be implemented using packages such as stacked packages (PoP), ball grid arrays (BGA), chip scale packages (CSP), plastic leaded chip carriers (PLCC), plastic dual in-line packages (PDIP), waffle packaged dies, wafer form dies, chip on board (COB), ceramic dual in-line packages (CERDIP), plastic metric quad flat packages (MQFP), thin quad flat packages (TQFP), small outline integrated circuits (SOIC), shrink small outline packages (SSOP), thin small outline packages (TSOP), system in package (SIP), multi-chip packages (MCP), wafer-level fabrication packages (WFP), wafer-level processing stacked packages (WSP), etc.

[0116] According to an embodiment of the present disclosure, the image sensor 2010 may be integrated with the processor 2020 in a single chip, or the image sensor 2010 and the processor 2020 may be separately integrated in different chips.

[0117] Furthermore, the computing system 2000 should be interpreted as any computing system that uses the image sensor 2010. For example, the computing system 2000 may include a digital camera, a mobile phone, a PDA, a PMP, a smart phone, and the like.

[0118] According to the present disclosure, an image sensing device may be provided to perform improved grid noise removal.

[0119] Although the image sensing device and its operating method are described with reference to specific embodiments, these are merely examples, the present disclosure is not limited thereto, and should be interpreted as having the widest scope based on the basic concepts disclosed in this specification. Those skilled in the art may implement unspecified embodiments by combining and replacing the disclosed embodiments, but these do not depart from the scope of the present disclosure. In addition, those skilled in the art may easily change or modify the embodiments disclosed in this specification, and it is obvious that these changes or modifications also fall within the scope of the present disclosure and the following claims. In addition, the embodiments may be combined to form additional embodiments.

[0120] CROSS-REFERENCE TO RELATED APPLICATIONS

[0121] This application claims priority to Korean Patent Application No. 10-2021-0074776, filed on Jun. 9, 2021, which is hereby incorporated by reference in its entirety.

Claims

1. An image processing device, comprising: a gain table generator that generates a gain table including gain table values ​​corresponding to a first resolution using pixel values ​​included in a region of interest included in the image; a gain table binner that converts the gain table into a target table including target table values ​​corresponding to a second resolution; as well as a calibrator that performs a calibration operation to remove noise included in the image based on the target table, wherein each of the gain table values ​​indicates a ratio of a first value to a second value, wherein the first value is an average pixel value of a first pixel corresponding to a gain region included in the gain table, and The second value is an average pixel value of second pixels in the region of interest whose color is the same as that of the first pixel.

2. The image processing device according to claim 1, wherein: The gain table includes a plurality of gain regions, and The gain table values ​​are values ​​corresponding to the plurality of gain regions, respectively.

3. The image processing device according to claim 2, wherein: The gain table values ​​of the respective gain regions are each obtained by dividing an average value of pixel values ​​having the same color as the pixel value corresponding to the gain region among pixel values ​​included in the region of interest by an average value of the pixel values ​​corresponding to the gain region.

4. The image processing device according to claim 3, wherein: The gain table generator generates a gain map of the entire image based on the gain table.

5. The image processing device according to claim 1, wherein: The gain table generator generates the gain table based on pixel values ​​other than pixel values ​​corresponding to pixels for phase detection auto focus (PDAF) in the region of interest. 6 . The image processing apparatus according to claim 1 , further comprising a gain table storage unit storing the gain table.

7. The image processing device according to claim 1, wherein: The first resolution is higher than the second resolution.

8. The image processing device according to claim 1, wherein: The number of the gain table values ​​is a positive integer multiple of the number of the target table values.

9. The image processing apparatus according to claim 1, wherein: The first resolution is a resolution corresponding to a hexadecimal pattern, and The second resolution is one of a resolution corresponding to a nine-cell pattern, a resolution corresponding to a four-element Bayer pattern, and a resolution corresponding to a Bayer pattern.

10. The image processing apparatus according to claim 1, wherein: The first resolution is a resolution corresponding to a nine-unit pattern, and The second resolution is a resolution corresponding to a quaternary Bayer pattern or a resolution corresponding to a Bayer pattern.

11. The image processing apparatus according to claim 1, wherein: The target table includes a plurality of calculation areas, and Each of the target table values ​​of each calculation area is an average value of the gain table values ​​corresponding to the calculation area.

12. The image processing apparatus according to claim 11, wherein: The number of the gain table values ​​corresponding to the plurality of calculation areas is determined based on the first resolution and the second resolution.

13. The image processing apparatus according to claim 1, wherein: The target table includes a plurality of calculation areas, and Each of the target table values ​​of each calculation area is a median value of the gain table values ​​corresponding to the calculation area.

14. The image processing apparatus according to claim 13, wherein: The number of the gain table values ​​corresponding to the calculation area is determined based on the first resolution and the second resolution.

15. The image processing apparatus according to claim 1, wherein: The calibrator applies the target table value to a plurality of pixel values ​​included in the image.

16. The image processing apparatus according to claim 1, wherein: The calibrator multiplies a plurality of pixel values ​​included in the image by the target table value in units of the second resolution.

17. The image processing apparatus according to claim 1, wherein: The noise includes grid noise caused by crosstalk between a plurality of pixels.

18. The image processing apparatus according to claim 1, wherein: The region of interest is a region having a predetermined size around one of the intersection points of a plurality of grid lines superimposed on the image.

19. An image sensing device, comprising: An image sensor that uses a plurality of pixels to obtain an image; as well as An image processor, which: generating a gain table including gain table values ​​corresponding to a first resolution using pixel values ​​included in a region of interest included in the image, converting the gain table to a target table including target table values ​​corresponding to a second resolution, the second resolution being lower than the first resolution, and removing noise included in the image based on the target table, wherein each of the gain table values ​​indicates a ratio of a first value to a second value, wherein the first value is an average pixel value of a first pixel corresponding to a gain region included in the gain table, and The second value is an average pixel value of second pixels in the region of interest whose color is the same as that of the first pixel.

20. The image sensing device according to claim 19, wherein: The target table value is generated using an average or a median value of at least two of the gain table values.

Images