Imaging System and Method for Reproducing Mosaic Bayer Images from Color Images

Abstract

An imaging system and a method for remosaicking a Bayer image from a color image, the imaging system comprising: a phase detection image sensor comprising a plurality of phase detection pixel units; and a processor configured to: interpolate a green image to obtain a full resolution interpolated green image comprising a defocused portion with artifacts and a focused portion with sharp image; low pass filter the full resolution interpolated green image to obtain a blurred image of the full resolution interpolated green image; combine the full resolution interpolated green image and the blurred image of the full resolution interpolated green image to obtain a corrected full resolution interpolated green image, wherein the artifacts of the defocused portion of the full resolution interpolated green image are removed and the sharp image of the focused portion of the full resolution interpolated green image is unaffected.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a phase-detection image sensor, and in particular to phase-detection image sensor remosaicing, and also to an imaging system and a method for remosaicking a Bayer image from a color image. BACKGROUND

[0002] Most electronic cameras have an autofocus capability. Recently, phase-detection autofocus (PDAF) has become popular. The autofocus function automatically focuses the camera on an object in a scene viewed by the camera. The autofocus can be fully automatic, such that the camera identifies an object in the scene and focuses on the object. In some cases, the camera can even decide which objects are more important than others, and then focus on the more important object. Alternatively, the autofocus can utilize user input that specifies which part or parts of the scene are of interest. Based on this, the autofocus function identifies objects in the one or more parts of the scene specified by the user, and focuses the camera on these objects. These functions are implemented by PDAF.

[0003] After autofocus, e.g. using PDAF, the captured image typically has in-focus and out-of-focus parts, because not all parts of the image are at the same distance from the camera. If PDAF is performed using a phase-detection (PD) image sensor, the out-of-focus parts can create artifacts. If low-pass filtering is used to remove the artifacts of the out-of-focus parts, the in-focus parts can be affected and become blurred. Therefore, there is a need for a PD image sensor that can present in-focus parts that are not affected and out-of-focus parts that have removed artifacts. SUMMARY

[0004] An imaging system is disclosed. The imaging system includes a phase detection image sensor and a processor. The phase detection image sensor includes a plurality of phase detection pixel units. Each phase detection pixel unit includes a first left phase detection pixel and a first right phase detection pixel. The first left phase detection pixel and the first right phase detection pixel are covered by a microlens. The plurality of phase detection pixel units includes green phase detection pixel units forming a green image, blue phase detection pixel units forming a blue image, and red phase detection pixel units forming a red image. The green image, the blue image, and the red image form a color image arranged in a raw Bayer pattern. The raw Bayer pattern includes the green phase detection pixel units, the blue phase detection pixel units, and the red phase detection pixel units. The processor is configured to: interpolate the green image to obtain a full resolution interpolated green image including a defocused portion having artifacts and a focused portion having a clear image; low pass filter the full resolution interpolated green image to obtain a blurred image of the full resolution interpolated green image; combine the full resolution interpolated green image and the blurred image of the full resolution interpolated green image to obtain a corrected full resolution interpolated green image; wherein the artifacts of the defocused portion of the full resolution interpolated green image are removed and the clear image of the focused portion of the full resolution interpolated green image is unaffected.

[0005] According to embodiments, each phase detection pixel unit further includes a second left phase detection pixel and a second right phase detection pixel. The first left phase detection pixel, the first right phase detection pixel, the second left phase detection pixel, and the second right phase detection pixel are covered by a microlens.

[0006] According to embodiments, the processor is further configured to: spatially variant convolve the full resolution interpolated green image with a bilateral kernel, wherein the bilateral kernel is a product of a spatial kernel and a range kernel.

[0007] According to embodiments, the spatial kernel includes a Gaussian function. According to embodiments, the range kernel is a function of the blurred image. The range kernel approaches one when a local variation of the blurred image is small, and the range kernel approaches zero when the local variation of the blurred image is large.

[0008] According to embodiments, the processor is further configured to: perform a joint least square fit of the full resolution interpolated green image and the blurred image within a scanning window including m x n points of the full resolution interpolated green image and the blurred image to obtain a corrected full resolution interpolated green image, wherein m and n are odd integers.

[0009] According to an embodiment, the joint least squares fitting comprises: determining a and b to obtain a minimum of a square of (aGi+b-Ti) within the scan window, where Gi is a value of the blurred image and Ti is a value of the full resolution interpolated green image; and determining the corrected full resolution interpolated green image as Cj=aGj+b using the determined a and b, where j indicates a center point of the scan window, and where Cj is a value of the corrected full resolution interpolated green image at the scan window and Gj is a value of the blurred image at the scan window.

[0010] According to an embodiment, the processor is further configured to correct a sensitivity of the green image, the blue image and the red image before interpolating the green image.

[0011] According to an embodiment, the processor is further configured to sharpen the corrected full resolution interpolated green image after obtaining the corrected full resolution interpolated green image.

[0012] According to an embodiment, the processor is further configured to interpolate the blue image and the red image based in part on the corrected full resolution interpolated green image and in part on the color images arranged in the original Bayer pattern; and form a Bayer image using portions of the interpolated blue image and the interpolated red image and the corrected full resolution interpolated green image.

[0013] A method for remosaicking a Bayer image from color images arranged in an original Bayer pattern is disclosed herein. The original Bayer pattern includes green pixel cells, blue pixel cells and red pixel cells. The method includes: interpolating a green image to obtain a full resolution interpolated green image including a defocused portion with artifacts and a focused portion with clear images; low pass filtering the full resolution interpolated green image to obtain a blurred image of the full resolution interpolated green image; combining the full resolution interpolated green image and the blurred image of the full resolution interpolated green image to obtain a corrected full resolution interpolated green image; wherein the artifacts of the defocused portion of the full resolution interpolated green image are removed and the clear images of the focused portion of the full resolution interpolated green image are unaffected.

[0014] According to an embodiment, the method for remosaicking a Bayer image from color images arranged in an original Bayer pattern further includes: performing a spatially variant convolution of the full resolution interpolated green image with a bilateral kernel, wherein the bilateral kernel is a product of a spatial kernel and a range kernel.

[0015] According to an embodiment, the spatial kernel includes a Gaussian function.

[0016] According to an embodiment, the range kernel is a function of the blurred image, the range kernel approaches one when a local variation of the blurred image is small, and the range kernel approaches zero when the local variation of the blurred image is large.

[0017] According to an embodiment, the method for remosaicking a Bayer image from color images arranged in a raw Bayer pattern further comprises performing a joint least squares fit of the full resolution interpolated green image and the blurred image within a scanning window containing m x n points of the full resolution interpolated green image and the blurred image to obtain a corrected full resolution interpolated green image, where m and n are odd integers.

[0018] According to an embodiment, the joint least squares fit comprises determining a and b to obtain a minimum of a square of (aGi + b - Ti) within the scanning window, where Gi is a value of the blurred image and Ti is a value of the full resolution interpolated green image, and determining the corrected full resolution interpolated green image as Cj = aGj + b using the determined a and b, where j indicates a center point of the scanning window, and where Cj is a value of the corrected full resolution interpolated green image at the scanning window and Gj is a value of the blurred image at the scanning window.

[0019] According to an embodiment, the method for remosaicking a Bayer image from color images arranged in a raw Bayer pattern further comprises correcting sensitivities of the green image, the blue image, and the red image prior to interpolating the green image.

[0020] According to an embodiment, the method for remosaicking a Bayer image from color images arranged in a raw Bayer pattern further comprises sharpening the corrected full resolution interpolated green image after obtaining the corrected full resolution interpolated green image.

[0021] According to an embodiment, the method for remosaicking a Bayer image from color images arranged in a raw Bayer pattern further comprises interpolating a blue image and a red image based in part on the corrected full resolution interpolated green image and in part on the color images arranged in the raw Bayer pattern, and forming a remosaicked Bayer image using portions of the interpolated blue image and the interpolated red image and the corrected full resolution interpolated green image.

[0022] This document discloses an imaging system comprising a plurality of green pixel units, a plurality of blue pixel units, a plurality of red pixel units, and a processor. Microlenses cover each green pixel unit, and each green pixel unit has four pixels. Microlenses cover each blue pixel unit, and each blue pixel unit has four pixels. Microlenses cover each red pixel unit, and each red pixel unit has four pixels. The plurality of green pixel units form a green image, the plurality of blue pixel units form a blue image, and the plurality of red pixel units form a red image; and the green image, the blue image, and the red image form a color image arranged in an original Bayer pattern, the original Bayer pattern including the green pixel units, the blue pixel units, and the red pixel units. The processor is configured to: interpolate the green image to obtain a full-resolution interpolated green image containing a defocused portion with artifacts and a focused portion with a sharp image; perform low-pass filtering on the full-resolution interpolated green image to obtain a blurred image of the full-resolution interpolated green image; combine the full-resolution interpolated green image and the blurred image of the full-resolution interpolated green image to obtain a corrected full-resolution interpolated green image; wherein the artifacts of the defocused portion of the full-resolution interpolated green image are removed, and the sharp image of the focused portion of the full-resolution interpolated green image is unaffected.

[0023] According to an embodiment, the processor is further configured to perform spatial variation convolution on the full-resolution interpolated green image and a bilateral kernel, wherein the bilateral kernel is the product of a spatial kernel and a range kernel.

[0024] According to an embodiment, the processor is further configured to: perform joint least-squares fitting of the full-resolution interpolated green image and the blurred image within a scanning window containing m×n points of the full-resolution interpolated green image and the blurred image to obtain a corrected full-resolution interpolated green image, where m and n are odd integers. Attached Figure Description

[0025] Non-limiting and non-exhaustive embodiments of the invention are described with reference to the following drawings, wherein, unless otherwise specified, the same reference numerals refer to the same parts throughout the various views.

[0026] FIG. 1A The pixel array of a PD image sensor with PDAF capability is shown.

[0027] FIG. 1B The cross-section of the microlens covering the left and right PD pixels is shown.

[0028] FIG. 1C The diagram shows a pixel array comprising multiple PD pixels and multiple microlenses covering the PD pixels.

[0029] FIG. 1D A cross-section of a microlens covering a first left PD pixel and a first right PD pixel is shown.

[0030] FIG. 2A A left image is shown separated from a right image to the left.

[0031] FIG. 2B A left image is shown overlapped with a right image.

[0032] FIG. 2C A left image is shown separated from a right image to the right.

[0033] FIG. 3A A pixel array comprising a plurality of PD pixels and a plurality of microlenses covering the PD pixels is shown.

[0034] FIG. 3B A set of four pixels of one color is shown.

[0035] FIG. 4 A Bayer pattern made up of each individual pixel of a PD image sensor is shown.

[0036] FIG. 5 A full resolution interpolated green image is shown with no missing green pixels.

[0037] FIG. 6A A defocused portion of an interpolated green image is shown.

[0038] FIG. 6B A focused portion of a full resolution interpolated green image is shown.

[0039] FIG. 7 A re-mosaicking method according to the present invention.

[0040] FIG. 8 A joint image filtering process according to the present invention is shown.

[0041] FIG. 9 A joint least squares fitting process according to the present invention is shown.

[0042] Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings. It will be understood that the elements of the drawings are merely schematic and that actual implementation of the present application can not necessarily be drawn to scale. For example, the dimensions of some of the elements in the figures can be exaggerated relative to other elements for the purpose of explanation.

[0043] BRIEF DESCRIPTION OF THE DRAWINGS

[0044] 100, 120, 300: pixel array;

[0045] 102, 114, 122, 322: microlenses;

[0046] 104: left phase detection pixel;

[0047] 106: right phase detection pixel;

[0048] 108, 110: light;

[0049] 112: image pixel;

[0050] 124: upper left PD pixel;

[0051] 126: upper right PD pixel;

[0052] 128: lower left PD pixel;

[0053] 130: lower right PD pixel;

[0054] 202: left image;

[0055] 204: right image;

[0056] 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316: PD pixel;

[0057] 324, 326, 328, 330, 344: pixel;

[0058] 340, 400: Bayer pattern;

[0059] 342: superpixel;

[0060] 500: green image;

[0061] 700: remosaic method;

[0062] 702, 704, 706, 708, 710, 712, 714, 716, 718, 720, 722: box;

[0063] 800: joint image filtering process;

[0064] 802, 902: full resolution interpolated green image;

[0065] 804, 904: blurred image;

[0066] 806: point P(x,y);

[0067] 808: point Q(x,y);

[0068] 810: spatial kernel;

[0069] 812: bilateral kernel;

[0070] 814, 908: corrected full resolution interpolated green image;

[0071] 816: range kernel;

[0072] 900: joint least squares fitting process;

[0073] 906: window. DETAILED DESCRIPTION

[0074] In the following description, numerous specific details are set forth to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details. In other instances, well-known materials or methods have not been described in detail in order to avoid obscuring the present application.

[0075] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable

[0076] FIG. 1A A pixel array 100 of a phase detection (PD) image sensor with PDAF capability is shown. In an embodiment, two pixels, e.g., a left phase detection (PD) pixel and a right PD pixel, are covered by a microlens. The phase difference between the left PD pixel and the right PD pixel is detected to perform autofocus. The pixel array 100 includes a plurality of PD pixels, a microlens 102 covers a left PD pixel 104 and a right PD pixel 106. The PD pixel 104 and the PD pixel 106 form a PD pixel cell. The pixel array 100 can also include a plurality of image pixels 112. Each image pixel has a microlens 114. An image pixel cannot be used as a PD pixel, but a PD pixel can be used as an image pixel.

[0077] FIG. 1BA cross-section of a microlens 102 covering a left PD pixel 104 and a right PD pixel 106 is shown. Light 108 from the left in the camera (not shown) is incident at the microlens 102 and directed to the right PD pixel 106. Light 110 from the right in the camera (not shown) is incident at the microlens 102 and directed to the left PD pixel 104. The left PD pixel 104 detects a left image of a PD image that is an image detected by a PD image sensor. The right PD pixel 106 detects a right image of the PD image. The PD pixel 104 and the PD pixel 106 can be identified as a first left PD pixel and a first right PD pixel, respectively.

[0078] In an embodiment, four pixels are covered by a microlens, in FIG. 1C is shown. FIG. 1C A pixel array 120 including a plurality of PD pixels and a plurality of microlenses covering the PD pixels is shown. A microlens 122 covers an upper left PD pixel 124, an upper right PD pixel 126, a lower left PD pixel 128, and a lower right PD pixel 130. The PD pixel 124, the PD pixel 126, the PD pixel 128, and the PD pixel 130 form a PD pixel cell. The pixel array 120 can also include a plurality of image pixels 112. Each image pixel has a microlens 114. An image pixel cannot be used as a PD pixel, but a PD pixel can be used as an image pixel.

[0079] A one-dimensional cross-section of the microlens 122 covering the PD pixel 124, the PD pixel 126, the PD pixel 128, and the PD pixel 130 is shown in FIG. 1D is shown. FIG. 1D Similar to FIG. 1B where the microlens 102 is replaced by the microlens 122, the PD pixel 104 is replaced by the PD pixel 124, and the PD pixel 106 is replaced by the PD pixel 126. The PD pixel 128 and the PD pixel 130 are not shown in the cross-section. The PD pixel 124, the PD pixel 126, the PD pixel 128, and the PD pixel 130 can be identified as a first left PD pixel, a first right PD pixel, a second left PD pixel, and a second right PD pixel, respectively.

[0080] FIG. 2A to FIG. 2C A left image of a PD image and a right image of a PD image formed by a left PD pixel and a right PD pixel, respectively, are shown. A PD image is referred to as an image detected by a PD image sensor before any processing. A PD image has a left image and a right image. FIG. 2A A left image 202 separated to the left from a right image 204 when the PD image is out of focus (e.g., front focus) is shown. FIG. 2B A left image 202 overlapping a right image 204 when the image is in focus is shown. FIG. 2CThe image shows a left image 202 that separates to the right from the right image 204 when the PD image is out of focus (e.g., back-defocus). The imaging lens of the camera (not shown) is moved to achieve this. FIG. 2B The image shown is the left image 202, which overlaps with the right image 204. However, since not all objects captured in the PD image are at the same distance from the camera, some parts of the PD image are in focus, and some parts are out of focus.

[0081] In this embodiment, all pixels in the pixel array can be PD pixels. In this embodiment, every four pixels can be covered by a microlens. FIG. 3A and FIG. 3B As shown in the figure. FIG. 3A A pixel array 300 comprising multiple PD pixels and multiple microlenses covering the PD pixels are shown. Microlens 322 covers four PD pixels, such as pixels 324, 326, 328, and 330. For example, PD pixels 301, 302, 303, and 304 may be blue pixels. PD pixels 305, 306, 307, and 308 may be green pixels. PD pixels 309, 310, 311, and 312 may also be green pixels. PD pixels 313, 314, 315, and 316 may be red pixels. FIG. 3B Pixel array 300 is shown to better illustrate the distribution of color pixels. It should be understood that other distributions of color pixels are also possible.

[0082] exist FIG. 3A In the illustrated embodiment, pixels 324, 326, 328, and 330 serve not only as PD pixels but also as image pixels. Furthermore, FIG. 3B All pixels in the pixel array 300 shown are used as PD pixels and image pixels. A pixel is used as a PD pixel when determining the phase difference between a pair of pixels covered by the same microlens. A pixel is used as an image pixel when it is individually read to form a PD image.

[0083] FIG. 3B This shows a group of four pixels representing one color. Furthermore, FIG. 3B A Bayer pattern 340 with 2×2 superpixels is shown. Each superpixel, such as superpixel 342, consists of 2×2 pixels (e.g., pixel 344) of the PD image sensor (which is both a PD pixel and an image pixel). To enhance the resolution of the PD image detected by the PD image sensor, the pixels contained in the superpixels are redistributed to form a Bayer pattern 400 composed of each individual pixel of the PD image sensor. FIG. 4As shown in the diagram. It's worth noting that Bayer pattern 400 is not a solid pixel array. In contrast, pixel array 300 is a solid pixel array. This process is called re-mosaicing the Bayer image. The re-mosaiced Bayer image of Bayer pattern 400 has green, blue, and red images. FIG. 3B Compared to the Bayer pattern 340 image with superpixels, the re-mosaiced Bayer image has double the resolution in both the x and y directions.

[0084] When re-painting the mosaic Bayer image, first interpolate the green image, for example from... FIG. 3B The Bayer 340 is used to obtain a full-resolution interpolated green image. This full-resolution interpolated green image is called the green image 500, which contains no lost green pixels. FIG. 5 As shown in the figure.

[0085] Included FIG. 3A to FIG. 3B A PD image sensor with a 300-pixel array can capture PD images that have some areas in focus and some areas out of focus. (Return to reference) FIG. 1D In the focusing section, the left PD pixel 124 and the right PD pixel 126 of the PD pixel unit detect the same light intensity because the left image 202 of the PD image overlaps with the right image 204 of the PD image. FIG. 2B As shown in the figure.

[0086] However, in the defocused portion, the left PD pixel 124 and the right PD pixel 126 detected different light intensities because the left image 202 of the PD image is offset from the right image 204 of the PD image. FIG. 2A or FIG. 2C As shown in the diagram, the left pixel of a PD pixel unit is adjacent to the right pixel of the same PD pixel unit. Therefore, in the defocused portion, two adjacent pixels (e.g., the left PD pixel and the right PD pixel) have different intensities. This produces artifacts in the captured PD image.

[0087] Interpolation algorithms for red, green, and blue images are available. These are not discussed in detail here, as they are not essential to the present invention. In this disclosure, it should be understood that a full-resolution interpolated green image is obtained or has been obtained. The full-resolution interpolated green image includes a defocused portion with artifacts and a focused portion with a sharp image. The full-resolution interpolated green image is then corrected using the method disclosed in this invention by removing artifacts while maintaining the sharp image. The mosaic Bayer image is then re-applied using the corrected full-resolution interpolated green image, provided that the artifacts have been removed and the sharp image is unaffected.

[0088] FIG. 6A This shows the out-of-focus portion of the full-resolution interpolated green image. In this example, the entire... FIG. 6AThis is a defocused image. The defocused portion shows artifacts in the image. Simply put, artifacts appear as white pixels between two black pixels. For example, the white pixel is the left PD pixel of a PD pixel unit, and the black pixels are the right PD pixel of the same PD pixel unit and the right PD pixel of an adjacent PD pixel unit. For comparison, FIG. 6B The image shown is the in-focus portion of the full-resolution interpolated green image, which displays a sharp image. In this example, the entire... FIG. 6B The image is in focus. FIG. 6B As shown, no artifacts appeared in the focused area.

[0089] FIG. 6A The artifacts in the defocused areas shown can be easily removed using a low-pass filter. However, a low-pass filter will... FIG. 6B The sharp image in the focused area shown is blurred, while artifacts are removed.

[0090] According to the present invention, in order to remove artifacts in the defocused portion while maintaining a clear image in the focused portion, in FIG. 7 As shown, a re-mosaic method 700 is disclosed. Box 702 indicates, for example, a method using... FIG. 3B The pixel array 300 shown is a PD image provided by a PD image sensor. Box 704 illustrates an optional step as sensitivity correction.

[0091] Return to reference FIG. 1D Ideally, light 108 from the left is directed to the right PD pixel 126, and light 110 from the right is directed to the left PD pixel 124. However, light 108 may be partially deflected to the left PD pixel 124, and light 110 may be partially deflected to the right PD pixel 126. To minimize the effect of light 108 on the left PD pixel 104 and light 110 on the right PD pixel 106, sensitivity correction 704 can be performed. The sensitivity of the green, blue, and red images can be corrected. Box 704 can produce sensitivity-corrected green, blue, and red images.

[0092] Box 706 illustrates the process of obtaining a full-resolution interpolated green image from the PD image in Box 702. Various algorithms are available. They are not discussed in detail here as they are not essential to the present invention. Box 708 illustrates the full-resolution interpolated green image after the full-resolution green interpolation in Box 706 is completed. The full-resolution interpolated green image includes portions with artifacts and portions with sharp images. Box 710 illustrates the low-pass filtering process.

[0093] Box 712 shows the blurred image after the low-pass filtering in box 710 is completed. The artifacts of the full-resolution interpolated green image in box 708 have been removed, but the sharp image in the full-resolution interpolated green image in box 708 has become blurred.

[0094] Box 714 illustrates a joint image filtering process or a joint least squares fitting process. Joint image filtering and / or joint least squares fitting together involve the full-resolution interpolated green image in box 708 and the blurred image in box 712. The full-resolution interpolated green image in box 706 and the blurred image in box 712 are combined or integrated in this process.

[0095] Box 716 shows the corrected full-resolution interpolated green image after the joint image filtering or joint least-squares fitting process in Box 714. Artifacts in the full-resolution interpolated green image in Box 708 have been removed, and the sharp image in the full-resolution interpolated green image in Box 708 is unaffected. Box 718 shows an optional step as green image sharpening. In this box, the corrected full-resolution interpolated green image in Box 716 can be further sharpened. Various sharpening algorithms including high-pass filtering are available.

[0096] Box 720 shows red and blue interpolation based in part on the corrected full-resolution interpolated green image of box 716 or the sharpened image of box 718. It should be understood that the interpolation in box 720 is not used to generate the full-resolution interpolated red and blue images. Instead, it is used to generate... FIG. 4 The Bayer image shown is from Bayer pattern 400. The Bayer image includes a green image with missing green pixels (two pixels missing in the Bayer pattern), a blue image with missing blue pixels (three pixels missing in the Bayer pattern), and a red image with missing red pixels (three pixels missing in the Bayer pattern). In contrast, the full-resolution interpolated image has no missing pixels. Box 722 shows the re-mosaiced Bayer image after the re-mosaicing method 700. The re-mosaiced Bayer image is formed using portions of the interpolated blue and red images in box 720 and the corrected interpolated green image in box 716.

[0097] In box 720, if box 704 is skipped, the red and blue images are interpolated partly based on the corrected full-resolution interpolated green image of box 716 or its sharpened image of box 718 and partly based on the color image arranged in the original Bayer pattern, said color image being the PD image shown in box 702. If box 704 is not skipped, the red and blue images are interpolated partly based on the sensitivity-corrected green, blue, and red images generated at box 704, rather than the PD image of box 702.

[0098] The re-mosaic method 700 can be executed by a processor of the imaging system. The imaging system includes a PD image sensor comprising a plurality of PD pixel units, each PD pixel unit including a first left PD pixel and a first right PD pixel, the first left PD pixel and the first right PD pixel being covered by microlenses. The plurality of PD pixel units includes green PD pixel units forming a green image, blue PD pixel units forming a blue image, and red PD pixel units forming a red image. The green image, blue image, and red image form a color image arranged in the original Bayer pattern, the original Bayer pattern including the green PD pixel units, blue PD pixel units, and red PD pixel units, the color image being the PD image shown in block 702.

[0099] The re-mosaic method 700 can be executed by the processor of the imaging system. The imaging system includes: multiple green pixel units, wherein microlenses cover each unit and each unit has four pixels, in... FIG. 3A and FIG. 3B As shown, there are multiple blue pixel units, each covered by a microlens and having four pixels; and multiple red pixel units, each covered by a microlens and having four pixels. Multiple green pixel units form a green image, multiple blue pixel units form a blue image, and multiple red pixel units form a red image. The green, blue, and red images form a color image arranged in an original Bayer pattern, which includes green, blue, and red pixel units, and the color image is similar to the PD image shown in box 702. In this embodiment, the pixel units are not PD pixel units, meaning that no phase difference between two pixels is detected.

[0100] In this embodiment, the microlens covers the pixel unit, and the pixel unit may have 2×2, 4×4, 8×8, or 16×16 pixels. In principle, the pixel unit may have m×m pixels, where m is any integer.

[0101] FIG. 8 Showing the invention FIG. 7 The joint image filtering process 800 is shown in box 714. Also... FIG. 7 The full-resolution interpolated green image 802 shown in box 708 is the target image. The target image is spatially mutated and convolved with the bilateral kernel 812 to generate the green image that is also... FIG. 7 Box 716 shows the corrected full-resolution interpolated green image 814. The bilateral kernel 812 is a joint bilateral function.

[0102] Also there FIG. 7The blurred image 804 shown in box 712 is used to generate the range kernel 816. For example, the blurred image 804 may be a low-pass filtered image of a full-resolution interpolated green image 802. The range kernel 816 is typically a window much smaller than the entire blurred image 804. For example, the range kernel 816 may be a 3×3 or 5×5 window. The range kernel 816 varies with the blurred image 804. When the local changes in the blurred image 804 are small, the range kernel 816 is close to one, and when the local changes in the blurred image 804 are large, the range kernel 816 is close to zero.

[0103] For example, kernel 816 within a 3×3 range at point Q(x,y)808 in the blurred image 804 can be expressed as:

[0104]

[0105] Where i = -1, 0, 1 and j = -1, 0, 1. R(x,y) is the kernel 816 within the range of Q(x,y)808 of the blurred image 804. I(x,y) is the blurred image 804 at Q(x,y)808, and I(x+iΔx,y+jΔy) is the blurred image 804 at the sampling points around Q(x,y)808 in the window. Δx and Δy are predetermined constants indicating the sample interval. σ is a constant.

[0106] When the local variation of the blurred image 804 (i.e., the square of the absolute value of I(x,y)-I(x+iΔx,y+jΔy)) is small, R(x,y)i,j approaches one, and when the local variation of the blurred image 804 (i.e., the square of the absolute value of I(x,y)-I(x+iΔx,y+jΔy)) is large, kernel 816 within the range approaches zero.

[0107] In this example, kernel 816 within a 3×3 range at Q(x,y)808 of the blurred image 804 can be expressed as:

[0108]

[0109] The range kernel 816 is multiplied by the spatial kernel 810 (e.g., a Gaussian function) to obtain the bilateral kernel 812 (e.g., a truncated Gaussian function at a point). Both the spatial kernel 810 and the bilateral kernel 812 can be the same window as the range kernel 816, for example, 3×3. The bilateral kernel 812 at point P(x,y)806 of the full-resolution interpolated green image 802 is the product of the spatial kernel 810 and the range kernel 816. The center of the spatial kernel 810 is aligned with the center of the range kernel 816, which corresponds to point Q(x,y)808 of the blurred image 804. Point Q(x,y)808 of the blurred image 804 is aligned with point P(x,y)806 of the full-resolution interpolated green image 802.FIG. 8 As shown in the figure.

[0110] In the spatial variation convolution between the full-resolution interpolated green image 802 and the bilateral kernel 812, the kernel (i.e., the bilateral kernel 812) changes or moves as the point P(x,y) 806 changes during the convolution. In this way, the sharp image (the in-focus portion of the interpolated green image) will not be smoothed by the bilateral kernel 812, while the out-of-focus portion of the interpolated green image with artifacts will be smoothed by the bilateral kernel 812.

[0111] The core 816, spatial core 810, and bilateral core 812 are not limited to 3×3 or 5×5 windows. They contain m×n windows, where m and n can be any numbers. Preferably, m and n are odd integers.

[0112] FIG. 9 This illustrates the joint least squares fitting process 900 according to the present invention, in FIG. 7 Box 714 in the diagram shows the process. A joint least-squares fit is performed between the full-resolution interpolated green image and the blurred image within the scanning window to obtain the corrected full-resolution interpolated green image. Also... FIG. 7 The full-resolution interpolated green image 902 shown in box 708 is the target image. Also... FIG. 7 The blurred image 904 shown in box 712 is the guide image. For example, blurred image 904 could be a low-pass filtered image of full-resolution interpolated green image 902. Window 906 encompasses the target image (full-resolution interpolated green image 902) and the guide image (blurred image 904). For example, window 906 contains 3×3 points of full-resolution interpolated green image 902, with values ​​T1, T2, T3, T4, T5, T6, T7, T8, and T9 respectively. T represents the target. Window 906 also contains 3×3 points of blurred image 904, with values ​​G1, G2, G3, G4, G5, G6, G7, G8, and G9 respectively. G represents the guide. A joint least-squares fitting process is performed to determine a and b, which will yield the minimum of the following squares.

[0113]

[0114] The above formula can be written as

[0115] a Gi+b-Ti,

[0116] Where Gi is the value of the i-th point of the blurred image 904 within the 3×3 window 906, and i is 1, 2, 3, ..., 9, and Ti is the value of the i-th point of the full-resolution interpolated green image 902 within the 3×3 window 906, and i is 1, 2, 3, ..., 9.

[0117] After determining a and b, the value at window 906 is also... FIG. 7 The corrected full-resolution interpolated green image 908 shown in box 716 is provided by the following

[0118] Cj = a Gj + b,

[0119] Where j indicates the center point of the 3×3 window 906 covering points with i (which are 1, 2, 3, ..., 9), Cj is the value of the center point of the 3×3 window 906 at the corrected full-resolution interpolated green image 908, and Gj is the value of the center point of the 3×3 window 906 at the blurred image 904. In this example, j is 5. The values ​​of all points in the corrected full-resolution interpolated green image 908 can be obtained through the scan window 906.

[0120] Window 906 is not limited to containing 3×3 points. It can cover m×n points of the full-resolution interpolated green image 902 and the blurred image 904, where m and n can be any numbers. Preferably, m and n are odd integers. In this way, the sharp image (the in-focus portion of the full-resolution interpolated green image) will not be smoothed through a joint least squares fitting process, while the out-of-focus portion of the full-resolution interpolated green image with artifacts will be smoothed through a joint least squares fitting process.

[0121] Although the invention has been described herein with respect to exemplary embodiments and the best mode for practicing the invention, it will be apparent to those skilled in the art that many modifications, improvements and sub-combinations of various embodiments, adaptations and variations can be made to the invention without departing from the spirit and scope of the invention.

[0122] The terminology used in the following claims should not be construed as limiting the invention to the specific embodiments disclosed in the specification and claims. In fact, the scope should be determined entirely by the appended claims, which should be interpreted according to the principles established by the claims. This specification and the drawings should therefore be considered illustrative rather than restrictive.

Claims

1. An imaging system comprising: a phase detection image sensor comprising a plurality of phase detection pixel cells, each phase detection pixel cell comprising a first left phase detection pixel and a first right phase detection pixel, the first left phase detection pixel and the first right phase detection pixel being covered by a micro lens; wherein the plurality of phase detection pixel cells comprises green phase detection pixel cells forming a green image, blue phase detection pixel cells forming a blue image, and red phase detection pixel cells forming a red image; and wherein the green image, the blue image, and red image form a color image arranged in a raw Bayer pattern, the raw Bayer pattern comprising the green phase detection pixel cells, the blue phase detection pixel cells, and the red phase detection pixel cells; a processor configured to: interpolate the green image to obtain a full resolution interpolated green image comprising a defocused portion having artifacts and a focused portion having a sharp image; low pass filter the full resolution interpolated green image to obtain a blurred image of the full resolution interpolated green image; combine the full resolution interpolated green image and the blurred image of the full resolution interpolated green image to obtain a corrected full resolution interpolated green image; wherein the artifacts of the defocused portion of the full resolution interpolated green image are removed and the sharp image of the focused portion of the full resolution interpolated green image is unaffected; spatially variant convolve the full resolution interpolated green image with a bilateral kernel, wherein the bilateral kernel is a product of a spatial kernel and a range kernel.

2. The imaging system of claim 1, wherein each phase detection pixel cell further comprises a second left phase detection pixel and a second right phase detection pixel, the first left phase detection pixel, the first right phase detection pixel, the second left phase detection pixel, and the second right phase detection pixel being covered by a micro lens.

3. The imaging system of claim 1, wherein the spatial kernel comprises a Gaussian function.

4. The imaging system of claim 1, wherein the range kernel is a function of the blurred image, the range kernel approaching one when a local variation of the blurred image is small, and the range kernel approaching zero when the local variation of the blurred image is large.

5. The imaging system of claim 1, wherein the processor is further configured to: correct a sensitivity of the green image, the blue image, and the red image prior to interpolating the green image.

6. The imaging system of claim 1, wherein the processor is further configured to: sharpen the corrected full resolution interpolated green image after obtaining the corrected full resolution interpolated green image.

7. The imaging system of claim 1, wherein the processor is further configured to: interpolate the blue image and the red image based in part on the corrected full resolution interpolated green image and in part on the color image arranged in the raw Bayer pattern; and forming a Bayer image using the interpolated blue image and the interpolated red image and portions of the corrected full resolution interpolated green image.

8. An imaging system comprising: a phase detection image sensor comprising a plurality of phase detection pixel cells, each phase detection pixel cell comprising a first left phase detection pixel and a first right phase detection pixel, the first left phase detection pixel and the first right phase detection pixel being covered by a microlens; wherein the plurality of phase detection pixel cells comprises green phase detection pixel cells forming a green image, blue phase detection pixel cells forming a blue image, and red phase detection pixel cells forming a red image; and wherein the green image, the blue image, and red image form a color image arranged in a raw Bayer pattern, the raw Bayer pattern comprising the green phase detection pixel cells, the blue phase detection pixel cells, and the red phase detection pixel cells; a processor configured to: interpolate the green image to obtain a full resolution interpolated green image comprising a defocused portion having artifacts and a focused portion having a clear image; low pass filter the full resolution interpolated green image to obtain a blurred image of the full resolution interpolated green image; combine the full resolution interpolated green image and the blurred image of the full resolution interpolated green image to obtain a corrected full resolution interpolated green image; wherein the artifacts of the defocused portion of the full resolution interpolated green image are removed and the clear image of the focused portion of the full resolution interpolated green image is unaffected; perform a joint least squares fit of the full resolution interpolated green image and the blurred image within a scan window of mxn points comprising the full resolution interpolated green image and the blurred image to obtain a corrected full resolution interpolated green image, where m and n are odd integers.

9. The imaging system of claim 8, wherein the joint least squares fit comprises: determine a and b to obtain a minimum of (aGi + b - Ti)2 within the scan window, where Gi is a value of the blurred image and Ti is a value of the full resolution interpolated green image; and determine the corrected full resolution interpolated green image as Cj = aGj + b using the determined a and b, where j indicates a center point of the scan window, and where Cj is a value of the corrected full resolution interpolated green image at the scan window and Gj is a value of the blurred image at the scan window.

10. The imaging system of claim 8, wherein each phase detection pixel cell further comprises a second left phase detection pixel and a second right phase detection pixel, the first left phase detection pixel, the first right phase detection pixel, the second left phase detection pixel, and the second right phase detection pixel being covered by a microlens.

11. The imaging system of claim 8, wherein the processor is further configured to: correct a sensitivity of the green image, the blue image, and the red image prior to interpolating the green image.

12. The imaging system of claim 8, wherein the processor is further configured to: sharpening the corrected full resolution interpolated green image after obtaining the corrected full resolution interpolated green image.

13. The imaging system of claim 8, wherein the processor is further configured to: interpolate the blue and red images based in part on the corrected full resolution interpolated green image and in part on the color images arranged in the original Bayer pattern; and form a Bayer image using the interpolated blue and red images and portions of the corrected full resolution interpolated green image.

14. A method for remosaicking a Bayer image from color images arranged in an original Bayer pattern, the original Bayer pattern including green pixel cells, blue pixel cells, and red pixel cells, the method comprising: interpolating a green image to obtain a full resolution interpolated green image including a defocused portion having artifacts and a focused portion having clear images; low pass filtering the full resolution interpolated green image to obtain a blurred image of the full resolution interpolated green image; combining the full resolution interpolated green image and the blurred image of the full resolution interpolated green image to obtain a corrected full resolution interpolated green image; wherein the artifacts of the defocused portion of the full resolution interpolated green image are removed and the clear images of the focused portion of the full resolution interpolated green image are unaffected; performing a spatially variant convolution of the full resolution interpolated green image with a bilateral kernel, wherein the bilateral kernel is a product of a spatial kernel and a range kernel.

15. The method of claim 14, wherein the spatial kernel includes a Gaussian function.

16. The method of claim 14, wherein the range kernel is a function of the blurred image, the range kernel approaching one when local variations of the blurred image are small and the range kernel approaching zero when the local variations of the blurred image are large.

17. The method of claim 14, further comprising: correcting sensitivities of the green, blue, and red images prior to interpolating the green image.

18. The method of claim 14, further comprising: sharpening the corrected full resolution interpolated green image after obtaining the corrected full resolution interpolated green image.

19. The method of claim 14, further comprising: interpolating the blue and red images based in part on the corrected full resolution interpolated green image and in part on the color images arranged in the original Bayer pattern; and using the interpolated blue and red images and portions of the corrected full resolution interpolated green image to form a remosaicked Bayer image.

20. A method for remosaicking a Bayer image from color images arranged in an original Bayer pattern, the original Bayer pattern including green pixel cells, blue pixel cells, and red pixel cells, the method comprising: ​ interpolating a green image to obtain a full resolution interpolated green image containing a defocused portion having artifacts and a focused portion having a sharp image; low pass filtering the full resolution interpolated green image to obtain a blurred image of the full resolution interpolated green image; combining the full resolution interpolated green image and the blurred image of the full resolution interpolated green image to obtain a corrected full resolution interpolated green image; wherein the artifacts of the defocused portion of the full resolution interpolated green image are removed and the sharp image of the focused portion of the full resolution interpolated green image is unaffected; performing a joint least squares fit of the full resolution interpolated green image and the blurred image within a scan window of mxn points containing the full resolution interpolated green image and the blurred image, where m and n are odd integers, to obtain a corrected full resolution interpolated green image.

21. The method of claim 20, wherein the joint least squares fit comprises: determining a and b to obtain a minimum of (aGi + b - Ti)2within the scan window, where Gi is a value of the blurred image and Ti is a value of the full resolution interpolated green image; and determining the corrected full resolution interpolated green image as Cj = aGj + b using the determined a and b, where j indicates a center point of the scan window, and where Cj is a value of the corrected full resolution interpolated green image at the scan window and Gj is a value of the blurred image at the scan window.

22. The method of claim 20, further comprising: correcting sensitivity of the green image, a blue image, and a red image prior to interpolating the green image.

23. The method of claim 20, further comprising: sharpening the corrected full resolution interpolated green image after obtaining the corrected full resolution interpolated green image.

24. The method of claim 20, further comprising: interpolating a blue image and a red image based in part on the corrected full resolution interpolated green image and in part on the color images in the original Bayer pattern arrangement; and forming a post-re-mosaiced Bayer image using the interpolated blue image and the interpolated red image and portions of the corrected full resolution interpolated green image.

25. An imaging system, comprising: a plurality of green pixel cells, a micro lens covering each cell and each cell having four pixels; a plurality of blue pixel cells, a micro lens covering each cell and each cell having four pixels; a plurality of red pixel cells, a micro lens covering each cell and each cell having four pixels; wherein the plurality of green pixel cells form a green image, the plurality of blue pixel cells form a blue image, and the plurality of red pixel cells form a red image; and wherein the green image, the blue image, and red image form color images in an original Bayer pattern arrangement, the original Bayer pattern including the green pixel cells, the blue pixel cells, and the red pixel cells; a processor configured to: interpolate a green image to obtain a full resolution interpolated green image containing a defocused portion having artifacts and a focused portion having a sharp image; low pass filter the full resolution interpolated green image to obtain a blurred image of the full resolution interpolated green image; combine the full resolution interpolated green image and the blurred image of the full resolution interpolated green image to obtain a corrected full resolution interpolated green image; wherein the artifacts of the defocused portion of the full resolution interpolated green image are removed and the sharp image of the focused portion of the full resolution interpolated green image is unaffected; perform a joint least squares fit of the full resolution interpolated green image and the blurred image within a scan window of mxn points containing the full resolution interpolated green image and the blurred image, where m and n are odd integers, to obtain a corrected full resolution interpolated green image. determine a and b to obtain a minimum of (aGi + b - Ti)2within the scan window, where Gi is a value of the blurred image and Ti is a value of the full resolution interpolated green image; and determine the corrected full resolution interpolated green image as Cj = aGj + b using the determined a and b, where j indicates a center point of the scan window, and where Cj is a value of the corrected full resolution interpolated green image at the scan window and Gj is a value of the blurred image at the scan window. interpolating the green image to obtain a full resolution interpolated green image containing defocused portions having artifacts and in-focus portions having sharp images; low pass filtering the full resolution interpolated green image to obtain a blurred image of the full resolution interpolated green image; combining the full resolution interpolated green image and the blurred image of the full resolution interpolated green image to obtain a corrected full resolution interpolated green image; wherein the artifacts of the defocused portions of the full resolution interpolated green image are removed and the sharp images of the in-focus portions of the full resolution interpolated green image are unaffected; spatially variant convolving the full resolution interpolated green image with a bilateral kernel, wherein the bilateral kernel is a product of a spatial kernel and a range kernel.

26. An imaging system comprising: a plurality of green pixel cells, each cell covered by a micro-lens and each cell having four pixels; a plurality of blue pixel cells, each cell covered by a micro-lens and each cell having four pixels; a plurality of red pixel cells, each cell covered by a micro-lens and each cell having four pixels; wherein the plurality of green pixel cells form a green image, the plurality of blue pixel cells form a blue image, and the plurality of red pixel cells form a red image; and wherein the green image, the blue image, and the red image form a color image arranged in a raw Bayer pattern, the raw Bayer pattern including the green pixel cells, the blue pixel cells, and the red pixel cells; a processor configured to: interpolate the green image to obtain a full resolution interpolated green image containing defocused portions having artifacts and in-focus portions having sharp images; low pass filter the full resolution interpolated green image to obtain a blurred image of the full resolution interpolated green image; combine the full resolution interpolated green image and the blurred image of the full resolution interpolated green image to obtain a corrected full resolution interpolated green image; wherein the artifacts of the defocused portions of the full resolution interpolated green image are removed and the sharp images of the in-focus portions of the full resolution interpolated green image are unaffected; perform a joint least squares fit of the full resolution interpolated green image and the blurred image within a scan window of m x n points containing the full resolution interpolated green image and the blurred image to obtain a corrected full resolution interpolated green image, wherein m and n are odd integers.

Images