Image signal processor
The image signal processor addresses the issue of reduced sampling rates in RGBW sensors by converting and synthesizing image data to improve image quality and reduce noise, particularly in low-light conditions.
Patent Information
- Application Number
- JP2023182180
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-07-14
- Filing Date
- 2023-10-24
- Publication Date
- 2025-01-24
AI Technical Summary
The use of RGBW image sensors in low-light conditions leads to reduced sampling rates for R, G, and B components, resulting in potential fake colors and the inability to utilize white pixels effectively for noise reduction during image processing.
An image signal processor that converts image data containing color and white pixel values into multiple images, extracts texture information, and synthesizes these images to generate a high-quality output by reducing noise and enhancing sharpness.
The processor effectively reduces noise and improves image quality by utilizing white pixels, resulting in sharper images with reduced noise and enhanced color accuracy.
Smart Images
Figure 2025013097000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an image signal processor for processing an image signal. [Background technology]
[0002] In recent years, as the functions of electronic devices have become more diverse, there has been an increasing demand for improved image capture functions using the electronic devices, which has led to a demand for techniques for improving the quality of images captured through the electronic devices.
[0003] Image sensors used in imaging devices have a structure in which color filters that transmit specific wavelength components of light (R, G, B) for each pixel are attached to the sensor surface. There are various types of color filter color arrangements, but the Bayer arrangement, which is made up of three types of filters that transmit only specific wavelengths of red (R), green (G), and blue (B), is widely used.
[0004] In recent years, as image sensor pixels become finer, the amount of light incident on each pixel decreases and the signal to noise ratio (SNR) increases, which can cause the performance of the image sensor to deteriorate rapidly. In order to deal with such low-light environments, increase the transmittance of the filter and achieve high sensitivity, image sensors that include white filters in addition to filters that transmit only specific wavelengths of light such as RGB (Red, Green, Blue) are becoming more widely used.
[0005] That is, a color pattern including a white channel that has a characteristic of being relatively resistant to signal noise compared to RGB channels is presented. The white channel is designed to receive light of the entire band compared to RGB channels that receive light of a specific band, so that it has a characteristic of having relatively little signal noise even when an image is captured in a low-light environment.
[0006] However, the RGBW array is a single-chip image sensor with R, G, B, and W filters arranged in a mosaic pattern, so when generating a color image, a re-mosaic process is required as color coding to generate RGB pixel values corresponding to each pixel.
[0007] However, an RGBW array including white pixels may have a lower sampling rate for the R, G, and B components compared to an RGB array. As a result, false colors may occur when the acquired data of the RGBW array is used in the process of generating a color image. For this reason, the white pixels are discarded in the re-mosaic processing stage, and the white pixels cannot be utilized in the subsequent stages. Summary of the Invention [Problem to be solved by the invention]
[0008] SUMMARY OF THE PREFERRED EMBODIMENTS An embodiment of the present invention provides an image signal processor that enables a pattern containing white pixels to utilize the white pixels for noise reduction and demosaicing to improve image quality. [Means for solving the problem]
[0009] An image signal processor according to an embodiment of the present invention may include a resolution conversion unit that generates two or more images based on image data including color pixel values and white pixel values, an element processing unit that extracts texture information using the converted color pixel values and the converted white pixel values included in each of the two or more images and generates two or more processed images based on the extracted texture information, a combination unit that combines the two or more processed images to generate a composite color difference image and a composite white image, and an image acquisition unit that generates an output image based on the image data, the composite color difference image, and the composite white image. Effect of the Invention
[0010] The embodiment of the present invention can reduce noise contained in image data and obtain a sharp image, thereby improving the quality of an image signal.
[0011] Furthermore, the embodiments of the present invention are for illustrative purposes only, and those skilled in the art can make various modifications, changes, substitutions, and additions within the technical spirit and scope of the appended claims, and such modifications, changes, etc. should be considered to fall within the scope of the following claims. [Brief description of the drawings]
[0012] [Figure 1] 1A to 1C are diagrams for explaining an example of an RGBW arrangement applied to an embodiment of the present invention. [Diagram 2] 2 is a diagram illustrating an example of image data input to an image signal processor according to an embodiment of the present invention; FIG. [Diagram 3] 1 is a diagram illustrating an image signal processor according to an embodiment of the present invention; [Figure 4] 4 is a diagram illustrating a method for reducing noise contained in image data and interpolating color differences according to the embodiment of FIG. 3; [Diagram 5] FIG. 4 is a diagram for explaining a detailed configuration of the image signal processor according to the embodiment of FIG. 3. [Figure 6] 6 is a diagram for explaining a detailed configuration and operation of a resolution conversion unit in FIG. 5. [Figure 7] 6 is a diagram for explaining a detailed configuration and operation of a resolution conversion unit in FIG. 5. [Figure 8] 6 is a diagram for explaining a detailed configuration of an element processing unit in FIG. 5. [Figure 9] 6 is a diagram for explaining a detailed configuration of an element processing unit in FIG. 5. [Figure 10] 6 is a diagram for explaining a detailed configuration of a synthesis unit in FIG. 5. FIG. [Figure 11]6 is a diagram for explaining a detailed configuration of an image acquisition unit in FIG. 5. FIG. [Figure 12] 12 is a diagram for explaining the operation of the color difference signal interpolation unit in FIG. 11. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Various embodiments will be described below with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to specific embodiments, and includes various modifications, equivalents, and / or alternatives of the embodiments. The embodiments of the present disclosure can provide various effects that are directly or indirectly recognizable by the present disclosure.
[0014] An image signal processor (ISP) according to an embodiment of the present invention can perform at least one image signal processing on image data to generate processed image data. The image signal processor can perform image signal processing for reducing noise on the image data and improving image quality, such as demosaicing, defect pixel correction, gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, and lens distortion correction.
[0015] The image signal processor may also generate an image file by compressing image data that has been subjected to image signal processing for improving image quality, or may restore image data from the image file. The image compression format may be a lossless format or a lossy format. As an example of the compression format, in the case of a still image, the JPEG (Joint Photographic Experts Group) format or the JPEG2000 format may be used. In the case of a moving image, a moving image file may be generated by compressing a plurality of frames according to the MPEG (Moving Picture Experts Group) standard.
[0016] The image data may be generated by an image sensing device that captures an optical image of a scene, although the scope of the present invention is not limited thereto. The image sensing device may include a pixel array including a plurality of pixels for sensing light incident from a scene, a control circuit for controlling the pixel array, and a readout circuit for converting analog pixel signals received from the pixel array into digital image data and outputting the digital image data. The present disclosure will be described on the assumption that the image data is generated by the image sensing device.
[0017] FIG. 1 is a diagram for explaining an example of an RGBW array applied to an embodiment of the present invention. 1, 16 pixels are shown arranged in a matrix including 4 rows and 4 columns. As an example, 16 pixels are the smallest unit of pixel arrays 10, 20, 30, and 16 pixels may be repeated in the row and column directions, but the scope of the present invention is not limited thereto.
[0018] The image sensing device according to the embodiment of the present invention may include color pixels such as red (R), green (G), and blue (B) and a white pixel (W). The color pixels (R, G, B) and the white pixel (W) may be arranged in various configurations as shown in FIG.
[0019] The color pixels (R, G, B) may be pixels to which light of a specific wavelength is incident through a corresponding color filter. The white pixel (W) is a pixel without a corresponding color filter, and may be referred to as a texture pixel, a black and white pixel, or a mono pixel. For example, the color pixels (R, G, B) can obtain a color pixel value by incident light of a specific color (e.g., red, green, blue, etc.) through a color filter that transmits only incident light of a specific wavelength. In contrast, the white pixel (W) can obtain a white pixel value by incident light (e.g., visible light) that is not limited to a specific color.
[0020] The pixel array 10 includes unit pixels (PX1) each including a 2x2 array of pixels, and each unit pixel (PX1) may include one color pixel (e.g., R, G, or B) and three white pixels (W).
[0021] The pixel array 20 includes unit pixels (PX2) each including a 2×2 array of pixels, and each unit pixel (PX2) may include two color pixels (e.g., R, G, or B) and two white pixels (W). In the case of the pixel array 20, the two color pixels included in each unit pixel (PX2) may be color pixels corresponding to the same color.
[0022] The pixel array 30 includes unit pixels (PX3) each including a 2×2 array of pixels, and each unit pixel (PX3) may include three color pixels (e.g., R, G, or B) and one white pixel (W). In the case of the pixel array 30, the three color pixels included in each unit pixel (PX3) may be color pixels that correspond to the same color.
[0023] 1 shows three examples in which color pixels (R, G, B) and white pixels (W) can be arranged, but the scope of the present disclosure is not limited thereto. For example, the pixel arrays 10, 20, 30 of the image sensor may include magenta, yellow, cyan, orange, and other pixels in addition to RGB. As another example, the pixel arrays 10, 20, 30 may be configured such that the color pixels are arranged at other possible positions (e.g., the lower right corner) instead of the upper right corner position (or the upper left corner and lower left corner positions) for each unit pixel (PX1, PX2, PX3).
[0024] As another example, the pixel arrays 10, 20, and 30 may be configured to include a different number of white pixels (W) for each unit pixel (PX1, PX2, and PX3), or white pixels (W) arranged at different positions. Thus, the pixel arrangements of the pixel arrays 10, 20, and 30 are not limited to those shown in the drawings, and may be configured in various forms to include white pixels (W) and color pixels.
[0025] FIG. 2 is a diagram illustrating an example of image data input to an image signal processor according to an embodiment of the present invention. 2 may show a form of image data 40 that is acquired and output by binning color pixels (R, G, B) and white pixels (W) by a binning unit 50 of an image signal processor. The embodiment of FIG. 2 will be described as an example in which the image data 40 is generated using pixel array 20 among pixel arrays 10, 20, and 30 of FIG. 1. The embodiment of FIG. 2 will be described as an example in which the binning unit 50 generates image data 40 in a half resolution mode, but the present disclosure is not limited thereto.
[0026] The binning unit 50 may output image data 40 having a smaller number of pixels (e.g., half) than the total number of pixels in the pixel array 20. The binning unit 50 may output one white pixel value 41 and one color pixel value (42, 43, or 44) for each unit pixel (PX2).
[0027] For example, the binning unit 50 may output a white pixel value 41 obtained by adding or averaging values obtained from two white pixels (W) included in each unit pixel (PX2). As another example, the binning unit 50 may output a red color pixel value 42 obtained by adding or averaging values obtained from two color pixels (R) included in each unit pixel (PX2). The binning unit 50 may output a green color pixel value 43 obtained by adding or averaging values obtained from two color pixels (G) included in each unit pixel (PX2). The binning unit 50 may output a blue color pixel value 44 obtained by adding or averaging values obtained from two color pixels (B) included in each unit pixel (PX2).
[0028] FIG. 2 has been described as an example in which the pixel array 20 includes an R, G, B, W arrangement and generates image data 40 in half resolution mode. However, according to an embodiment, even if the pixel array includes a pixel array having another arrangement, the type of image data generated according to the half resolution mode or full resolution mode can be determined.
[0029] FIG. 3 is a diagram illustrating an image signal processor according to an embodiment of the present invention. Referring to FIG. 3, an image signal processor (ISP) may include a binning unit 50, a resolution conversion unit 100, an element processing unit 200, a synthesis unit 300, and an image acquisition unit 400.
[0030] The binning unit 50 may bin color pixels (R, G, B) and white pixels (W) in image data received from an image sensor (not shown) to generate image data 40 including color pixel values and white pixel values. Here, the binning unit 50 may correspond to the binning unit 50 already described in FIG. 2 above.
[0031] The resolution conversion unit 100 may generate two or more images having different sizes based on the image data 40. According to an embodiment, the resolution conversion unit 100 may include a first resolution conversion unit that converts the resolution of the image data 40 and obtains a first image having a reduced size compared to the image data 40. The resolution conversion unit 100 may also include a second resolution conversion unit that converts the resolution of the first image and obtains a second image having a reduced size compared to the first image. For example, the first image may have a size reduced to 1 / 2 compared to the image data 40, and the second image may have a size reduced to 1 / 4 compared to the image data 40.
[0032] The element processor 200 can extract texture information using the converted color pixel values and the converted white pixel values included in each of the two or more images. Here, the texture may refer to the shape or contour of an object on an image. The element processor 200 can generate two or more images for synthesis and color difference values based on the texture information, and perform noise reduction processing on the converted white pixel values.
[0033] According to an embodiment, the element processing unit 200 may include a first element processing unit that generates a first processed image corresponding to the first image using a first converted color pixel value and a first converted white pixel value included in the first image. Also, the element processing unit 200 may include a second element processing unit that generates a second processed image corresponding to the second image using a second converted color pixel value and a second converted white pixel value included in the second image.
[0034] The synthesizer 300 may synthesize two or more processed images to generate a synthetic color difference image and a synthetic white image. For example, the synthesizer 300 may generate a synthetic color difference image and a synthetic white image based on a first processed image and a second processed image.
[0035] The image acquisition unit 400 can acquire an output image based on the composite color difference image, the composite white image, the processed image from the element processing unit 200, and the white pixel value 41 included in the image data 40. The white pixel value and the color difference value that have undergone noise reduction processing by the element processing unit 200 are synthesized by the synthesis unit 300 and input to the image acquisition unit 400, so that the image acquisition unit 400 can generate an output image in which noise is reduced and color differences are corrected compared to the image data 40.
[0036] A specific method in which the image signal processor (ISP) reduces noise contained in image data and interpolates color differences using the resolution conversion unit 100, element processing unit 200, synthesis unit 300, and image acquisition unit 400 will be described later with reference to the flowchart of FIG. 4 and examples of FIG. 5 to FIG. 12.
[0037] FIG. 4 is a diagram illustrating a method for reducing noise contained in image data and interpolating color differences according to the embodiment of FIG. In step S10, the binning unit 50 can generate image data (e.g., 40 in FIG. 2) including color pixel values (e.g., 42 to 44 in FIG. 2) and a white pixel value (e.g., 41 in FIG. 2) using the color pixels (e.g., R, G, B) and white pixels (W) of the pixel array 20.
[0038] In step S20, an image signal processor (ISP) (e.g., resolution converter 100) may generate two or more images having different sizes based on image data (e.g., 40). The resolution converter 100 may reduce the resolution of the image data to obtain a plurality of images. As the resolution of the image data is converted by the resolution converter 100, the two or more images may include converted color pixel values and converted white pixel values.
[0039] In step S30, an image signal processor (ISP) (eg, element processing unit 200) can extract texture information using the transformed color pixel values and the transformed white pixel values contained in each of the two or more images.
[0040] In step S40, the image signal processor (ISP) (e.g., element processor 200) can generate a synthesis image based on the extracted texture information, calculate color difference values, and perform noise reduction processing on the converted white pixel values. In this way, the element processor 200 can generate two or more processed images corresponding to the two or more images, respectively.
[0041] In step S50, an image signal processor (ISP) (eg, the combiner 300) may combine two or more processed images to obtain a combined color difference image and a combined white image.
[0042] The synthesis unit 300 may generate a synthetic color difference image and a synthetic white image based on the first processed image for some regions where texture is above a certain level (e.g., image regions where edges or thin lines exist) and the second processed image for other regions (e.g., flat image regions or gradation image regions). For example, the synthesis unit 300 may perform multi-scale processing on two or more processed images to generate a synthetic color difference image and a synthetic white image. Here, the multi-scale processing may mean selecting and processing a processed image with a low resolution (second processed image) for a flat image region where texture does not exist and selecting and processing a processed image with a high resolution (first processed image) for an image region where texture exists, based on processed images having different sizes.
[0043] In step S60, an image signal processor (ISP) (e.g., image acquisition unit 400) may acquire an output image based on the composite color difference image, the composite white image, and the white pixel value (e.g., 41 in FIG. 2). For example, the image acquisition unit 400 may interpolate a color difference signal based on the composite color difference image, the composite white image, and the white pixel value. The image acquisition unit 400 may also generate an edge enhancement signal based on the composite color difference image, the composite white image, and the white pixel value to enhance the sharpness of overly blurred image regions.
[0044] The image signal processor (ISP) can obtain an output image that is noise-reduced and not excessively blurred by using a white pixel value together with a composite image obtained based on images having different sizes. Unlike the present disclosure, when performing multi-scale processing on an image (e.g., a color image) that does not use a white pixel value, certain areas in the composite image may be excessively blurred. For example, multi-scale processing on an image may mean selecting and synthesizing an unscaled image for an area where texture is present, and a scaled image (or an image simply enlarged after reducing the resolution) for an area where texture is not present. However, if the image contains noise, it may be determined that no texture is present in an image area where texture is present, and a scaled image may be selected and synthesized. That is, if the image contains noise, it may be difficult to determine whether texture is present or not due to the noise, and therefore an area where texture is present may be excessively blurred.
[0045] However, in accordance with the present disclosure, an image signal processor (ISP) can generate an edge enhancement signal based on the original non-noise-reduced white pixel values, thereby minimizing blur in the output image by restoring overly blurred (or undesirably blurred) areas in the composite image.
[0046] A specific example in which components included in an image signal processor (ISP) generate an output image will now be described. FIG. 5 is a diagram for explaining a detailed configuration of the image signal processor according to the embodiment of FIG.
[0047] 5, an image signal processor (ISP) may receive binned image data 40 via a binning unit 50. Here, the image data 40 may correspond to the image data 40 shown in FIG.
[0048] The image signal processor (ISP) converts the resolution of the image data 40 through the resolution converter 110, and can generate a first image (HRI1) having a reduced size compared to the image data 40. Here, the first image (HRI1) can include R, G, B, and W pixel values having a reduced size compared to the image data 40. For example, the first image (HRI1) can have a half size compared to the image data 40.
[0049] The image signal processor (ISP) converts the resolution of the first image (HRI1) through the resolution converter 120 to generate a second image (HRI2) having a reduced size compared to the first image (HRI1). The second image (HRI2) may include R, G, B, and W pixel values having a reduced size compared to the first image (HRI1). For example, the second image (HRI2) may have a quarter size compared to the image data 40.
[0050] Similarly, the image signal processor (ISP) can obtain a third image (HRI3) having a reduced size compared to the second image (HRI2) through the resolution converter 130, can obtain a fourth image (HRI4) having a reduced size compared to the third image (HRI3) through the resolution converter 140, can obtain a fifth image (HRI5) having a reduced size compared to the fourth image (HRI4) through the resolution converter 150, and can obtain a sixth image (HRI6) having a reduced size compared to the fifth image (HRI5) through the resolution converter 160. The third image (HRI3) can have a size of 1 / 8 compared to the image data 40, the fourth image (HRI4) can have a size of 1 / 16 compared to the image data 40, the fifth image (HRI5) can have a size of 1 / 32 compared to the image data 40, and the sixth image (HRI6) can have a size of 1 / 64 compared to the image data 40.
[0051] The image signal processor (ISP) may generate a processed image (PI1) using color pixel values and white pixel values included in the image data 40 through the element processor 210. Here, the element processor 210 may extract texture information based on the color pixel values and white pixel values included in the image data 40 and output an image for synthesis. The element processor 210 may perform noise reduction on the white pixel value based on the extracted texture information. The element processor 210 may also calculate color differences for the color pixel values and white pixel values based on the extracted texture information. An image set including the image for synthesis, the noise-reduced white image, and color difference values output by the element processor 210 may be understood as a processed image (PI1). For example, the processed image (PI1) may have the same size as the image data 40.
[0052] The image signal processor (ISP) may generate a processed image (PI2) using the converted color pixel values and the converted white pixel values included in the first image (HRI1) through the element processor 220. The element processor 220 may extract texture information based on the converted color pixel values and the converted white pixel values included in the first image (HRI1) and output a synthesis image. The element processor 220 may perform noise reduction on the white pixel values based on the extracted texture information. The element processor 220 may also calculate color differences for the color pixel values and the white pixel values based on the extracted texture information. An image set including the synthesis image, the noise-reduced white image, and the color difference values output by the element processor 220 may be understood as a processed image (PI2). For example, the processed image (PI2) may have a size half that of the image data 40.
[0053] The image signal processor (ISP) may generate a processed image (PI3) using the converted color pixel values and the converted white pixel values included in the second image (HRI2) through the element processor 230. The element processor 230 may extract texture information based on the converted color pixel values and the converted white pixel values included in the second image (HRI2) and output a synthesis image. The element processor 230 may perform noise reduction on the white pixel values based on the extracted texture information. The element processor 230 may also calculate color differences for the color pixel values and the white pixel values based on the extracted texture information. An image set including the synthesis image, the noise-reduced white image, and the color difference values output by the element processor 230 may be understood as a processed image (PI3). For example, the processed image (PI3) may have a size that is 1 / 4 of the image data 40.
[0054] Similarly, the image signal processor (ISP) can generate a processed image (PI4) corresponding to the third image (HRI3) through the element processor 240, can generate a processed image (PI5) corresponding to the fourth image (HRI4) through the element processor 250, can generate a processed image (PI6) corresponding to the fifth image (HRI5) through the element processor 260, and can generate a processed image (PI7) corresponding to the sixth image (HRI6) through the element processor 270. Here, the processed image (PI4) can have a size of 1 / 8 compared to the image data 40, the processed image (PI5) can have a size of 1 / 16 compared to the image data 40, the processed image (PI6) can have a size of 1 / 32 compared to the image data 40, and the processed image (PI7) can have a size of 1 / 64 compared to the image data 40.
[0055] The image signal processor (ISP) can generate a composite image (SI1) by combining the processed image (PI2) and the composite image (SI2) through the combining unit 310. Here, an image set including a composite color difference image and a composite white image output by the combining unit 310 can be understood as the composite image (SI1). The composite image (SI1) can have a size half that of the image data 40.
[0056] The image signal processor (ISP) can generate a composite image (SI2) by combining the processed image (PI3) and the composite image (SI3) through the combining unit 320. Here, an image set including a composite color difference image and a composite white image output by the combining unit 320 can be understood as the composite image (SI2). The composite image (SI2) can have a size that is 1 / 4 of the image data 40.
[0057] Similarly, the image signal processor (ISP) may generate a composite image (SI3) by combining the processed image (PI4) and the composite image (SI4) through the combining unit 330, may generate a composite image (SI4) by combining the processed image (PI5) and the composite image (SI5) through the combining unit 340, and may generate a composite image (SI5) by combining the processed image (PI6) and the processed image (PI7) through the combining unit 350. Here, the composite image (SI3) may have a size that is 1 / 8 of the image data 40, the composite image (SI4) may have a size that is 1 / 16 of the image data 40, and the composite image (SI5) may have a size that is 1 / 32 of the image data 40.
[0058] The image signal processor (ISP) can generate an RGB image using the white pixel value 41, the processed image (PI1), and the composite image (SI1) included in the image data 40 via the image acquisition unit 400. The image signal processor (ISP) can convert the RGBW image to RGB via the image acquisition unit 400 and obtain the output image 500.
[0059] Although the embodiment of FIG. 5 has been illustrated and described as an image signal processor (ISP) including six resolution conversion units 110-160, seven element processing units 210-270, and five synthesis units 310-350, this is merely one example and is not intended to limit the scope of the present disclosure. Each component shown in FIG. 5 will be described in more detail with reference to FIGS. 6 to 12, which will be described later.
[0060] FIG. 6 is a diagram for explaining a detailed configuration of the resolution conversion unit 110 in FIG. Referring to FIG. 6, the resolution converter 110 may include a Bayer resolution converter 113 and a White resolution converter 114 .
[0061] The Bayer resolution converter 113 converts the resolution of the image data 40 for the Bayer image 111, and can output color images (R, G, B images) having a reduced size compared to the image data 40. For example, the R, G, B image (corresponding to the first image (HRI1)) output by the Bayer resolution converter 113 can have half the size compared to the Bayer image 111. For example, half the size may mean reducing the number of pixels by half and reducing the data storage capacity by half.
[0062] The white resolution converter 114 converts the resolution of the white image 112 of the image data 40, and outputs a white image having a reduced size compared to the image data 40. For example, the white image output by the white resolution converter 114 may have a half size compared to the image data 40.
[0063] Here, the positions of the Bayer image 111 and the white image 112 of the image data 40 input to the resolution conversion unit 110 may be misaligned. Therefore, the Bayer resolution conversion unit 113 may perform interpolation so that the position of the central pixel of the white image whose size has been reduced coincides with the center of gravity of each of the central pixels of the R, G, and B images whose size has been reduced. That is, the Bayer resolution conversion unit 113 may calculate the interpolation coefficients of each of the R, G, and B pixels based on the white image 112 in order to prevent a blurring phenomenon from occurring in the white image 112 for obtaining texture information when the Bayer image 111 and the white image 112 are downscaled. For example, the Bayer resolution conversion unit 113 may perform interpolation using a method such as bilateral interpolation, bicubic interpolation, or bilinear interpolation so that the centers of gravity of the R, G, and B images coincide with the centers of gravity of the white image. Here, the interpolation processing method of the R, G, and B images for matching the centers of gravity is not limited thereto.
[0064] FIG. 7 is a diagram for explaining a detailed configuration of the resolution conversion unit 120 in FIG. Referring to FIG. 7, the resolution converter 120 may include a red (R) resolution converter 121, a green (G) resolution converter 122, a blue (B) resolution converter 123, and a white (W) resolution converter .
[0065] Here, the red resolution converter 121 may convert the resolution of the first image (HRI1) of the resolution converter 110, i.e., the red image, and output a red image having a reduced size compared to the image data 40. For example, the red resolution converter 121 may average pixel values obtained from four red pixels (R) and output a red image having a reduced size.
[0066] The green resolution converter 122 may convert the resolution of the first image (HRI1) of the resolution converter 110, i.e., the green image, and output a green image having a reduced size compared to the image data 40. For example, the green resolution converter 122 may average pixel values obtained from four green pixels (G) and output a green image having a reduced size.
[0067] In addition, the blue resolution converter 123 may convert the resolution of the first image (HRI1) of the resolution converter 110, i.e., the blue image, and output a blue image having a reduced size compared to the image data 40. For example, the blue resolution converter 123 may average pixel values obtained from four blue pixels (B) and output a blue image having a reduced size.
[0068] The white resolution converter 124 may convert the resolution of the first image (HRI1) of the resolution converter 110, i.e., the white image, and output a white image having a reduced size compared to the image data 40. For example, the white resolution converter 124 may average pixel values obtained from four white pixels (W) and output a white image having a reduced size.
[0069] For example, the R, G, B, W image output by the resolution converter 120, that is, the second image (HRI2), may have a size half that of the input image (first image (HRI1)).
[0070] The remaining resolution conversion units 130 to 160 have the same configuration as resolution conversion unit 120, except for the size of the images input and output compared to resolution conversion unit 120. Therefore, redundant descriptions of the detailed configurations and operations of resolution conversion units 130 to 160 will be omitted.
[0071] FIG. 8 is a diagram for explaining a detailed configuration of element processing unit 210 in FIG. Referring to FIG. 8, the element processing unit 210 may include a texture determining unit 213 , a noise reducing unit 214 , and a color difference calculating unit 215 .
[0072] The texture determination unit 213 can determine the texture based on the Bayer image 211 and the white image 212 of the image data 40 and generate texture information (TI) and a synthesis image (T). The Bayer image 211 and the white image 212 input to the texture determination unit 213 may be images similar to the Bayer image 111 and the white image 112 shown in FIG.
[0073] Here, the texture information (TI) may include information on whether the Bayer image 211 and the white image 212 are flat image regions or gradation image regions, information on whether edges or thin lines exist, and, if thin lines exist, information on the direction of the thin lines. As an example, the texture determination unit 213 may preferentially determine the texture information (TI) on the white image 212. For example, the texture determination unit 213 may first determine the texture information (TI) in the white image 212, and if no texture information exists in the white image 212, determine the texture information (TI) in the Bayer image 211.
[0074] In addition, when the image acquisition unit 400 described in FIG. 4 and FIG. 5 performs a multi-scale process, the ratio of two images to be synthesized can be adjusted based on the texture information (TI). Thus, the texture determination unit 213 can convert the texture information (TI) into a synthesis image (T) for the synthesis operation of the image acquisition unit 400 and provide it to the image acquisition unit 400. Here, the synthesis image (T) is an image used when synthesizing two images having different resolutions. As an example, two images can be synthesized using an alpha blending method. For example, an output image having a resolution that matches the high-resolution image can be generated by performing an interpolation process using linear interpolation on a low-resolution image. Although the synthesis of two images using the alpha blending method has been described as an example, the embodiment of the present invention is not limited thereto, and various methods for synthesizing two images are possible. The synthesis image (T) can be defined as an image (value) in which data representing the amplitude or synthesis ratio of a texture is calculated for each pixel. For example, the compositing image (T) can be used to combine two images by increasing the proportion of the high-resolution image in boundary areas with edges or texture, and by increasing the proportion of the low-resolution image in flat areas without edges or texture.
[0075] The noise reduction unit 214 can perform noise reduction processing on pixel values of the white image 212 based on the texture information (TI) to generate a noise-reduced white image (W_NR1). The noise reduction unit 214 can perform different noise reduction processing on a part of the region where the texture is equal to or higher than a certain level and on another part of the region where the texture is lower than the certain level based on the texture information (TI).
[0076] For example, the noise reduction unit 214 may perform noise reduction by obtaining a weighted average of surrounding pixels around a target pixel (a central pixel at the center of an input image unit) along the texture direction (the direction of the edge or thin line) in some regions where the texture is at or above a certain level (e.g., an image region where an edge or thin line exists) based on the texture information (TI). Also, the noise reduction unit 214 may perform noise reduction by obtaining an average of surrounding pixels around a target pixel in other regions where the texture is below a certain level (e.g., a flat image region or a gradation image region) based on the texture information (TI).
[0077] Furthermore, the color difference calculation unit 215 can calculate a color difference value (B_CD1) between the Bayer image 211 and the white image 212 based on the texture information (TI).
[0078] For example, the color difference calculation unit 215 may calculate a color difference along a texture direction (a direction of an edge or a thin line) in a certain region (e.g., an image region where an edge or a thin line exists) where the texture is equal to or higher than a certain level based on the texture information (TI). Also, the color difference calculation unit 215 may calculate a color difference using surrounding pixels around a target pixel in another certain region (e.g., a flat image region or a gradation image region) where the texture is less than a certain level based on the texture information (TI).
[0079] The synthesis image (T), the noise-reduced white image (W_NR1), and the color difference value (B_CD1) generated by the element processor 210 can be transmitted to the image acquisition unit 400 as a processed image (PI1).
[0080] FIG. 9 is a diagram for explaining a detailed configuration of element processing unit 220 in FIG. Referring to FIG. 9, the element processing unit 220 may include a texture determining unit 223 , a noise reducing unit 224 , and a color difference calculating unit 225 .
[0081] Here, the texture determination unit 223 can determine the texture based on the first image (HRI1) 221 (red image, green image, blue image) and the white image 222 of the resolution conversion unit 110, and generate a synthesis image (T) and texture information (TI).
[0082] 4 and 5, when the synthesizing unit 310 synthesizes two input images, the ratio of the synthesized image can be adjusted based on the texture information (TI). Thus, the texture determination unit 223 can convert the texture information (TI) into a synthesis image (T) for the synthesis operation of the synthesizing unit 310 and provide it to the synthesizing unit 310.
[0083] The noise reduction unit 224 can perform noise reduction processing on pixel values of the white image 222 based on the texture information (TI) to generate a noise-reduced white image (W_NR2). The noise reduction unit 224 can perform different noise reduction processing on a part of the region where the texture is equal to or higher than a certain level and on another part of the region where the texture is lower than the certain level based on the texture information (TI).
[0084] For example, the noise reduction unit 224 may perform noise reduction by obtaining a weighted average of surrounding pixels of a target pixel along the texture direction (the direction of the edge or thin line) in some regions (e.g., image regions where edges or thin lines exist) where the texture is at or above a certain level based on the texture information (TI). Also, the noise reduction unit 224 may perform noise reduction by obtaining an average of surrounding pixels around the target pixel in other regions (e.g., flat image regions or gradation image regions) where the texture is below a certain level based on the texture information (TI).
[0085] In addition, the color difference calculation unit 225 can calculate a color difference value (RGB_CD1) between the RGB image and the white image 222 based on the texture information (TI).
[0086] For example, the color difference calculation unit 225 may calculate a color difference along the texture direction (the direction of the edge or thin line) for some regions (e.g., image regions with edges or thin lines) where the texture is at or above a certain level based on the texture information (TI). Also, the color difference calculation unit 225 may calculate a color difference using surrounding pixels around the target pixel for other regions (e.g., flat image regions or gradation image regions) where the texture is below a certain level based on the texture information (TI). For example, the color difference may be calculated by dividing the average of the R (or G, B) pixels by the average of the white pixels.
[0087] The synthesis image (T), the noise-reduced white image (W_NR2), and the color difference value (RGB_CD1) generated by the element processor 220 can be transferred to the synthesis unit 310 as a processed image (PI2).
[0088] The remaining element processing units 230 to 270 have the same configuration as element processing unit 220, except for the size of the images input and output compared to element processing unit 220, so duplicated explanations of the detailed configurations and operations of element processing units 230 to 270 will be omitted.
[0089] FIG. 10 is a diagram for explaining a detailed configuration of the synthesis unit 310 in FIG. Referring to FIG. 10, the synthesis unit 310 may include a color difference synthesis unit 311 and a white (W) image synthesis unit 312 .
[0090] The chrominance synthesis unit 311 may generate a synthetic chrominance image (RGB_CD3) by synthesizing a chrominance value (RGB_CD1) and a chrominance value (RGB_CD2) based on a synthesis image (T) applied from the element processor 220. Here, the chrominance value (RGB_CD1) may mean a chrominance value for an RGB image (corresponding to a processed image (PI2)) applied from the element processor 220. And, the chrominance value (RGB_CD2) may mean a chrominance value for an RGB image (corresponding to a synthetic image (SI2)) having a reduced resolution applied from the synthesis unit 320.
[0091] The white image synthesizer 312 may generate a synthesized white image (W_NR4) by synthesizing the white image (W_NR2) and the white image (W_NR3) based on the synthesis image (T) applied from the element processor 220. Here, the white image (W_NR2) may refer to a noise-reduced white image (corresponding to the processed image (PI2)) applied from the element processor 220. And the white image (W_NR3) may refer to a white image with a reduced resolution (corresponding to the synthesized image (SI2)) applied from the synthesizer 320.
[0092] For example, the color difference synthesis unit 311 and the white image synthesis unit 312 can synthesize two input images by increasing the ratio of a reduced image (e.g., synthesis image (SI2)) in flat image regions or gradation image regions based on the synthesis image (T) and increasing the ratio of a non-reduced input image (e.g., processing image (PI2)) in image regions where edges or thin lines exist. This allows some regions where the texture is above a certain level to maintain clarity, and the remaining regions where the texture is below a certain level to reduce noise.
[0093] The remaining synthesis units 320 to 350 have the same configuration as synthesis unit 310, except for the size of the images input and output compared to synthesis unit 310. Therefore, a duplicated description of the detailed configuration and operation of synthesis units 300 to 350 will be omitted.
[0094] Fig. 11 is a diagram for explaining a detailed configuration of the image acquisition unit in Fig. 5. Fig. 12 is a diagram for explaining the operation of the color difference signal interpolation unit in Fig. 11.
[0095] Referring to FIG. 11, the image acquisition unit 400 may include a color difference synthesis unit 401, a white (W) image synthesis unit 402, an edge enhancement signal generation unit 403, a moire region determination unit 404, a color difference signal interpolation unit 405, and an image generation unit 406.
[0096] The chrominance synthesis unit 401 may generate a synthetic chrominance image (RGB_CD4) by synthesizing the chrominance value (B_CD1) applied from the element processor 210 and the synthetic chrominance image (RGB_CD3) applied from the synthesizer 310 based on the synthesis image (T) generated by the element processor 210. Here, since the chrominance value (B_CD1), which is one input of the chrominance synthesis unit 401, is a Bayer image, the synthetic chrominance image (RGB_CD4) may also be a Bayer image.
[0097] The white image synthesis unit 402 can generate a synthetic white image (W_NR5) by synthesizing the noise-reduced white image (W_NR1) applied from the element processing unit 210 and the synthetic white image (W_NR4) applied from the synthesis unit 310 based on the synthesis image (T) generated by the element processing unit 210.
[0098] For example, the color difference synthesis unit 401 and the white image synthesis unit 402 can synthesize two input images by increasing the ratio of a reduced image (e.g., a synthesis image (SI1)) in flat image areas or gradation image areas based on the synthesis image (T), and increasing the ratio of a non-reduced input image (e.g., a processing image (PI1)) in image areas where edges or thin lines are present.
[0099] The edge enhancement signal generator 403 performs edge enhancement processing on the white image (W1) (corresponding to 112 in FIG. 6) of the image data 40 and the composite white image (W_NR5) applied from the white image composition unit 402, and can output a white image (W2) after the edge enhancement processing is completed. That is, before the image generation unit 406 generates the output RGB image, the edge enhancement signal generator 403 performs edge enhancement processing to increase the contrast, thereby making it possible to further clarify the image texture of the output RGB image.
[0100] As an example, the edge enhancement signal generator 403 may extract high frequency components from the white image (W1) of the original image data 40 in which noise has not been reduced, and may generate a texture enhancement component by filtering out frequencies higher than a preset frequency. The edge enhancement signal generator 403 may then add the texture enhancement component to a composite white image (W_NR5) in which noise reduction has been completed, to generate a white image (W2) in which edge enhancement has been completed. In addition, the edge enhancement signal generator 403 may perform edge enhancement by combining the composite white image (W_NR5) in which noise reduction has been completed and the original white image (W1) in which noise has not been reduced, thereby enhancing the sharpness of areas that have become excessively blurred in the noise reduction process.
[0101] The moire region determination unit 404 can determine a moire region based on the white image (W1) of the image data 40 and the composite white image (W_NR5) applied from the image composition unit 402, and generate a moire determination signal (MDS). Here, moire can refer to an uneven image that occurs when a subject having a repeated pattern is photographed, and the pattern causes interference of high frequency components in an area similar to the spatial frequency of the image sensor. The moire region determination unit 404 can detect a moire region from the white image (W1) and the composite white image (W_NR5), and generate a moire determination signal (MDS) corresponding to the presence or absence of detection.
[0102] As an example, the area where color moiré (false color) occurs may be an area where there is a texture with a higher frequency than the Nyquist frequency corresponding to the sampling interval of the color pixels. In the case of white pixels, since the number of pixels is smaller than the number of output pixels, there may be a frequency area where it is not possible to confirm the presence or absence of a texture of a high frequency component. In the embodiment of the present invention, since the white image is input with the same resolution (number of pixels) as the output image, there is no need to interpolate the white image. Therefore, by referring to the white image, it is possible to determine the presence or absence of a high frequency texture up to the Nyquist frequency of the output image. That is, the moiré area determination unit 404 can determine the presence or absence of a high frequency component around each pixel of the white image and determine whether or not it is an area where color moiré occurs.
[0103] Here, the moiré region determination unit 404 may determine a color moiré region by using a method of calculating the intensity of the amplitude of high-frequency components using a Fourier transform, a Hadamard transform, a high-pass filter, or the like. As another example, a method may be used in which the absolute difference values of adjacent pixels in the horizontal and vertical directions are added with a central pixel as a reference, and the difference value is judged based on the magnitude of the added value. As another example, the central pixel may be used as a reference, and the absolute difference values of the upper and lower pixels may be compared with the absolute difference values of the left and right pixels, and an average value of the smaller pixel pair (upper and lower pixels or left and right pixels) may be calculated. Then, the absolute difference value between the calculated average value and the pixel value of the central pixel is calculated, and the calculated absolute difference value is compared with the amount of noise, and if it is greater than the amount of noise, it can be judged as a color moiré region. The method of determining a moiré region is not limited to this, and other methods may be used.
[0104] The color difference signal interpolation unit 405 can interpolate the composite white image (W_NR5) and the composite color difference image (RGB_CD4) based on the moire determination signal (MDS) and the composite image (T) and output an interpolated RGB image (IRGB). Here, the composite white image (W_NR5) that has the same resolution as the output image and does not require interpolation can be used as an input to the color difference signal interpolation unit 405. The color difference signal interpolation unit 405 can interpolate the color difference between the red image (or green image, blue image) and the white image based on the composite white image (W_NR5) and generate an interpolated RGB image (IRGB).
[0105] 12A, the color difference signal interpolation unit 405 may determine a direction in a white image when a color moire region is not found based on the moire determination signal (MDS), and perform interpolation of R color difference, G color difference, and B color difference based on the direction. According to an embodiment, the interpolation of R color difference, G color difference, and B color difference may be linear interpolation, but the scope of the present invention is not limited thereto.
[0106] In contrast, referring to FIG. 12B, the chrominance signal interpolation unit 405 may interpolate chrominance when it is a color moiré region based on the moiré judgment signal (MDS). According to an embodiment, the chrominance interpolation may be performed using filtering that considers edge information, such as a guided filter. That is, it may be assumed that each chrominance image (chrominance(x)) is a texture that is linearly transformed using arbitrary coefficients (a, b) of a white image W(x). The coefficients (a, b) of the linear transformation may be calculated by linear regression to calculate a pixel value of a central pixel, thereby performing the interpolation. For example, the pixel values of the white image and the chrominance image of a plurality of surrounding pixels present around the central pixel may be used to calculate the linear regression. The chrominance image (chrominance(x)) may be determined using the coefficients (a, b) calculated by linear regression.
[0107] In addition, the image generating unit 406 may generate an output RGB image based on the white image (W2) on which edge enhancement processing has been completed and the RGB image (IRGB) on which interpolation has been completed. As an example, the image generating unit 406 may generate an output RGB image by multiplying the RGB image (IRGB) on which interpolation processing has been completed by a pixel value of the white image (W2) on which edge enhancement processing has been completed. In this way, the RGB image generating unit 406 may generate an output RGB image on which color distortion has been reduced and noise has been reduced.
Claims
1. a resolution conversion unit that generates two or more images based on image data including color pixel values and white pixel values; an element processing unit that extracts texture information using the converted color pixel values and the converted white pixel values included in each of the two or more images, and generates two or more processed images based on the extracted texture information; a synthesis unit for synthesizing the two or more processed images to generate a synthetic color difference image and a synthetic white image; an image capture unit for generating an output image based on the image data, the composite color difference image, and the composite white image; an image signal processor including:
2. The resolution conversion unit is a first resolution converter for converting a resolution of the image data to generate a first image having a reduced size compared to the image data; 2. The image signal processor of claim 1, further comprising: a second resolution converter for converting a resolution of the first image to generate a second image having a reduced size compared to the first image.
3. The first resolution conversion unit is a Bayer resolution converter for converting a resolution of the image data with respect to a Bayer image and outputting a first red image, a first green image, and a first blue image each having a size reduced compared to the Bayer image; 3. The image signal processor of claim 2, further comprising: a white resolution converter for converting a resolution of the image data for a white image and outputting a first white image having a reduced size compared to the white image.
4. The Bayer resolution conversion unit 4. The image signal processor of claim 3, wherein the first red image, the first green image, and the first blue image are interpolated so that a central pixel of each of the first red image, the first green image, and the first blue image has a center of gravity coincident with a central pixel of the first white image.
5. The second resolution conversion unit is a red resolution converter for converting a resolution of the first red image and outputting a second red image having a reduced size compared to the first red image; a green resolution converter that converts a resolution of the first green image and outputs a second green image having a reduced size compared to the first green image; a blue resolution converter that converts a resolution of the first blue image and outputs a second blue image having a reduced size compared to the first blue image; 4. The image signal processor of claim 3, further comprising: a white resolution converter configured to convert a resolution of the first white image and output a second white image having a reduced size compared to the first white image.
6. The element processing unit includes: a first element processing unit that extracts texture information using the image data and generates a first processed image based on the extracted texture information; a second element processing unit for generating a second processed image corresponding to the first image using the first transformed color pixel values and the first transformed white pixel values included in the first image; and a third element processing unit for generating a third processed image corresponding to the second image using second transformed color pixel values and second transformed white pixel values contained in the second image.
7. The first element processing unit is a first texture determination unit that determines a texture based on a Bayer image and a white image of the image data, and outputs texture information and a synthesis image obtained by converting the texture information; a first noise reduction unit that performs noise reduction processing on pixel values of the white image based on the texture information to generate a noise-reduced third white image; The image signal processor of claim 6 , further comprising: a first color difference calculator for calculating a first color difference value between the Bayer image and the white image based on the texture information.
8. The texture information is 8. The image signal processor of claim 7, further comprising at least one of information regarding whether the Bayer image and the white image are flat image regions or gradient image regions, information regarding the presence of edges or thin lines, and direction information of the thin lines.
9. The first noise reduction unit is 8. The image signal processor according to claim 7, wherein different noise reduction processes are performed on a portion of the region where the texture is equal to or higher than a certain level and on another portion of the region where the texture is lower than the certain level, based on the texture information.
10. The second element processing unit is a second texture determination unit that determines a texture based on the first converted color pixel value and the first converted white pixel value, and outputs texture information and a synthesis image obtained by converting the texture information; a second noise reduction unit for performing a noise reduction process on the first converted white pixel values based on the texture information to generate a noise-reduced fourth white image; and a second color difference calculator for calculating second color difference values of the first transformed color pixel values and the first transformed white pixel values based on the texture information.
11. The synthesis unit is a first synthesis unit for synthesizing a second processed image and a second synthesized image among the two or more processed images to generate a first synthesized image; 11. The image signal processor of claim 10, further comprising: a second combiner configured to combine a third processed image and a third combined image among the two or more processed images to generate the second combined image.
12. The first synthesis unit is a first color difference synthesis unit for generating a first synthesized color difference image by synthesizing the second color difference value and a third color difference value of the second image based on the synthesis image; 12. The image signal processor of claim 11, further comprising: a first white image synthesis unit for generating a first synthesized white image by synthesizing the fourth white image and a fifth white image included in the second synthesized image based on the synthesis image.
13. The first synthesis unit is 13. The image signal processor of claim 12, wherein a ratio of the second color difference value and the third color difference value is adjusted before being combined, and a ratio of the fourth white image and the fifth white image is adjusted before being combined.
14. The image acquisition unit includes: a second color difference synthesis unit for generating a second synthetic color difference image by synthesizing a first color difference value and the first synthetic color difference image based on the synthetic image; a second white image composition unit that generates a second composite white image by combining a third white image from the first element processing unit and the first composite white image based on the composite image; an edge enhancement signal generating unit for generating a sixth white image on which edge enhancement processing has been completed by combining the white image of the image data and the second composite white image; a moire region determination unit that determines a moire region based on the white image of the image data and the second composite white image, and generates a moire determination signal; a color difference signal interpolation unit that interpolates the second composite color difference image and the second composite white image based on the moire determination signal and the composite image, and outputs an interpolated RGB image; 13. The image signal processor of claim 12, further comprising: an image generator for generating an output RGB image based on the sixth white image and the interpolated RGB image.
15. The image acquisition unit includes:
15. The image signal processor of claim 14, wherein the first color difference values and the first composite color difference image are combined by adjusting a ratio thereof, and the third white image and the first composite white image are combined by adjusting a ratio thereof.
16. The edge enhancement signal generation unit 15. The image signal processor of claim 14, further comprising: extracting high frequency components from the white image to generate a texture-enhanced component; and adding the generated texture-enhanced component to the second composite white image to generate the sixth white image.
17. The moire region determination unit 15. The image signal processor according to claim 14, further comprising: determining whether or not there is a high frequency component around each pixel of the white image and the second composite white image, and determining whether or not there is a region where color moire occurs.
18. The color difference signal interpolation unit performing an interpolation using filtering when a color moire region is determined based on the moire determination signal; 15. The image signal processor of claim 14, further comprising: determining a direction in the second composite white image when the region is other than the color moire region based on the moire determination signal; and performing R, G, and B color difference interpolation based on the determined direction.
19. The image generating unit includes:
15. The image signal processor of claim 14, further comprising: multiplying the interpolated RGB image by pixel values of the sixth white image to generate the output RGB image.
20. 2. The image signal processor of claim 1, further comprising a binning unit for binning color pixels and white pixels in said image data to generate said color pixel values and said white pixel values.