Imaging system and method for re-mosaicing bayer image from color image

By using interpolation, low-pass filtering and combined least squares fitting technology in the phase detection autofocus system, the artifact problem in the image sensor is solved, the clarity of the focus part and the removal of the defocus part is achieved, and the image is improved overall quality and resolution.

CN119031243AActive Publication Date: 2024-11-26OMNIVISION TECHNOLOGIES INC
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202410646015.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-24
Filing Date
2024-05-23
Publication Date
2024-11-26
Estimated Expiration
2044-05-23

AI Technical Summary

Technical Problem

When using phase detection autofocus technology, images captured by the image sensor usually contain focus and defocus parts, resulting in artifacts appearing, and low-pass filtering to remove artifacts affects the clarity of the focus parts.

Method used

An imaging system is designed, including multiple phase detection pixel units and processors, and the artifacts of the defocused part are removed and the clarity of the focus part is maintained through technologies such as interpolation, low-pass filtering and combined least squares fitting. A corrected full resolution interpolated image is formed and a remosaic Bayer image is generated based on this.

Benefits of technology

Effectively remove artifacts from the defocused part, maintain the sharpness of the focus part, and improve the overall quality and resolution of the image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119031243A_ABST
    Figure CN119031243A_ABST
Patent Text Reader

Abstract

An imaging system and a method for re-mosaicing a Bayer image from a color image, the imaging system comprising: a phase detection image sensor comprising a plurality of phase detection pixel units; and a processor configured to: interpolate the green image to obtain a full resolution interpolated green image comprising a defocused portion with artifacts and a focused portion with a sharp image; performing low-pass filtering on the full-resolution interpolation green image to obtain a blurred image of the full-resolution interpolation green image; the full-resolution interpolated green image and the blurred image of the full-resolution interpolated green image are combined to obtain a corrected full-resolution interpolated green image wherein artifacts of the defocused portion of the full-resolution interpolated green image are removed and the clear image of the focused portion of the full-resolution interpolated green image is unaffected.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a phase detection image sensor, and more particularly to phase detection image sensor remosaicing, and also to an imaging system and a method for remosaicing a Bayer image from a color image. Background Art

[0002] Most electronic cameras have autofocus capabilities. Recently, phase-detection autofocus (PDAF) has become popular. The autofocus function automatically focuses the camera on objects in the scene that the camera is looking at. Autofocus can be fully automatic, so that the camera recognizes objects in the scene and focuses on the objects. In some cases, the camera can even decide which objects are more important than other objects, and then focus on the more important objects. Alternatively, autofocus can utilize user input that specifies which part or parts of the scene are of interest. Based on this, the autofocus function recognizes objects in one or more parts of the scene specified by the user, and focuses the camera on these objects. These functions are implemented by PDAF.

[0003] After autofocusing, such as using PDAF, the captured image usually has in-focus and out-of-focus parts because not all parts of the image are at the same distance from the camera. If a phase-detection (PD) image sensor is used to perform PDAF, the out-of-focus parts may produce artifacts. If low-pass filtering is used to remove the artifacts of the defocused parts, the in-focus parts may be affected and become blurred. Therefore, a PD image sensor that can present an unaffected in-focus part and an artifact-removed defocused part is needed. Summary of the invention

[0004] An imaging system is disclosed herein. The imaging system includes a phase detection image sensor and a processor. The phase detection image sensor includes a plurality of phase detection pixel units. Each phase detection pixel unit includes a first left phase detection pixel and a first right phase detection pixel. The first left phase detection pixel and the first right phase detection pixel are covered by a microlens. The plurality of phase detection pixel units include a green phase detection pixel unit that forms a green image, a blue phase detection pixel unit that forms a blue image, and a red phase detection pixel unit that forms a red image. The green image, the blue image, and the red image form a color image arranged in an original Bayer pattern. The original Bayer pattern includes the green phase detection pixel unit, the blue phase detection pixel unit, and the red phase detection pixel unit. The processor is configured to: interpolate the green image to obtain a full-resolution interpolated green image including a defocused portion with artifacts and a focused portion with a clear image; low-pass filter the full-resolution interpolated green image to obtain a blurred image of the full-resolution interpolated green image; combine the full-resolution interpolated green image and the blurred image of the full-resolution interpolated green image to obtain a corrected full-resolution interpolated green image; wherein the artifacts of the defocused portion of the full-resolution interpolated green image are removed and the clear image of the focused portion of the full-resolution interpolated green image is not affected.

[0005] According to an embodiment, each phase detection pixel unit further comprises a second left phase detection pixel and a second right phase detection pixel. The first left phase detection pixel, the first right phase detection pixel, the second left phase detection pixel and the second right phase detection pixel are covered by a microlens.

[0006] According to an embodiment, the processor is further configured to: perform a spatially-variant convolution on the full-resolution interpolated green image with a bilateral kernel, wherein the bilateral kernel is a product of a spatial kernel and an in-range kernel.

[0007] According to an embodiment, the spatial kernel comprises a Gaussian function. According to an embodiment, the in-range kernel is a function of the blurred image. When the local variation of the blurred image is small, the in-range kernel approaches one, and when the local variation of the blurred image is large, the in-range kernel approaches zero.

[0008] According to an embodiment, the processor is further configured to: perform a joint least squares fitting of the full-resolution interpolated green image and the blurred image within a scanning window of m×n points including the full-resolution interpolated green image and the blurred image to obtain a corrected full-resolution interpolated green image, where m and n are odd integers.

[0009] According to an embodiment, the joint least squares fitting includes: determining a and b to obtain the minimum value of the square of (aGi+b-Ti) within the scanning window, where Gi is the value of the blurred image and Ti is the value of the full-resolution interpolated green image; and using the determined a and b to determine the corrected full-resolution interpolated green image as Cj=aGj+b, where j indicates the center point of the scanning window, and where Cj is the value of the corrected full-resolution interpolated green image at the scanning window, and Gj is the value of the blurred image at the scanning window.

[0010] According to an embodiment, the processor is further configured to correct the sensitivity of the green image, the blue image and the red image before interpolating the green image.

[0011] According to an embodiment, the processor is further configured to: sharpen the corrected full-resolution interpolated green image after obtaining the corrected full-resolution interpolated green image.

[0012] According to an embodiment, the processor is further configured to: interpolate the blue image and the red image based in part on the corrected full-resolution interpolated green image and in part on the color image arranged in the original Bayer pattern; and form a Bayer image using the interpolated blue image and the interpolated red image and a portion of the corrected full-resolution interpolated green image.

[0013] A method for re-mosaicing a Bayer image from a color image arranged in an original Bayer pattern is disclosed. The original Bayer pattern includes a green pixel unit, a blue pixel unit, and a red pixel unit. The method includes: interpolating a green image to obtain a full-resolution interpolated green image including a defocused portion with artifacts and a focused portion with a clear image; low-pass filtering the full-resolution interpolated green image to obtain a blurred image of the full-resolution interpolated green image; combining the full-resolution interpolated green image and the blurred image of the full-resolution interpolated green image to obtain a corrected full-resolution interpolated green image; wherein the artifacts of the defocused portion of the full-resolution interpolated green image are removed, and the clear image of the focused portion of the full-resolution interpolated green image is not affected.

[0014] According to an embodiment, the method for re-mosaicing a Bayer image from a color image arranged in an original Bayer pattern further comprises: performing a spatially-variant convolution of the full-resolution interpolated green image with a bilateral kernel, wherein the bilateral kernel is the product of a spatial kernel and an in-range kernel.

[0015] According to an embodiment, the spatial kernel comprises a Gaussian function.

[0016] According to an embodiment, the in-range kernel is a function of the blurred image, and when the local variation of the blurred image is small, the in-range kernel is close to one, and when the local variation of the blurred image is large, the in-range kernel is close to zero.

[0017] According to an embodiment, the method for re-mosaicing a Bayer image from a color image arranged in an original Bayer pattern also includes: performing a joint least squares fitting of the full-resolution interpolated green image and the blurred image within a scanning window of m×n points including the full-resolution interpolated green image and the blurred image to obtain a corrected full-resolution interpolated green image, where m and n are odd integers.

[0018] According to an embodiment, the joint least squares fitting includes: determining a and b to obtain the minimum value of the square of (aGi+b-Ti) within the scanning window, where Gi is the value of the blurred image and Ti is the value of the full-resolution interpolated green image; and using the determined a and b to determine the corrected full-resolution interpolated green image as Cj=aGj+b, where j indicates the center point of the scanning window, and where Cj is the value of the corrected full-resolution interpolated green image at the scanning window, and Gj is the value of the blurred image at the scanning window.

[0019] According to an embodiment, the method for re-mosaicing a Bayer image from a color image arranged in an original Bayer pattern further comprises: correcting the sensitivity of the green image, the blue image and the red image before interpolating the green image.

[0020] According to an embodiment, the method for re-mosaicing a Bayer image from a color image arranged in an original Bayer pattern further comprises: after obtaining the corrected full-resolution interpolated green image, sharpening the corrected full-resolution interpolated green image.

[0021] According to an embodiment, the method for re-mosaicing a Bayer image from a color image arranged in an original Bayer pattern also includes: interpolating a blue image and a red image based in part on the corrected full-resolution interpolated green image and in part on the color image arranged in the original Bayer pattern; and forming a re-mosaiced Bayer image using the interpolated blue image and the interpolated red image and a portion of the corrected full-resolution interpolated green image.

[0022] The present invention discloses an imaging system, which includes a plurality of green pixel units, a plurality of blue pixel units, a plurality of red pixel units, and a processor. A microlens covers each green pixel unit and each green pixel unit has four pixels. A microlens covers each blue pixel unit and each blue pixel unit has four pixels. A microlens covers each red pixel unit and each red pixel unit has four pixels. Wherein the plurality of green pixel units form a green image, the plurality of blue pixel units form a blue image, and the plurality of red pixel units form a red image; and wherein the green image, the blue image, and the red image form a color image arranged in an original Bayer pattern, and the original Bayer pattern includes the green pixel unit, the blue pixel unit, and the red pixel unit. The processor is configured to: interpolate the green image to obtain a full-resolution interpolated green image including a defocused portion with artifacts and a focused portion with a clear image; low-pass filter the full-resolution interpolated green image to obtain a blurred image of the full-resolution interpolated green image; combine the full-resolution interpolated green image and the blurred image of the full-resolution interpolated green image to obtain a corrected full-resolution interpolated green image; wherein the artifacts of the defocused portion of the full-resolution interpolated green image are removed and the clear image of the focused portion of the full-resolution interpolated green image is not affected.

[0023] According to an embodiment, the processor is further configured to: perform a spatially-variant convolution on the full-resolution interpolated green image with a bilateral kernel, wherein the bilateral kernel is a product of a spatial kernel and an in-range kernel.

[0024] According to an embodiment, the processor is further configured to: perform a joint least squares fitting of the full-resolution interpolated green image and the blurred image within a scanning window of m×n points including the full-resolution interpolated green image and the blurred image to obtain a corrected full-resolution interpolated green image, where m and n are odd integers. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Non-limiting and non-exhaustive embodiments of the present invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

[0026] Figure 1A Shown is a pixel array of a PD image sensor with PDAF capability.

[0027] Figure 1B A cross section of a microlens covering the left PD pixel and the right PD pixel is shown.

[0028] Figure 1C A pixel array including a plurality of PD pixels and a plurality of micro lenses covering the PD pixels is shown.

[0029] Figure 1D A cross section of a microlens covering a first left PD pixel and a first right PD pixel is shown.

[0030] Figure 2A The left image is shown separated to the left from the right image.

[0031] Figure 2B A left image is shown overlapped with a right image.

[0032] Figure 2C The left image is shown separated to the right from the right image.

[0033] Figure 3A A pixel array including a plurality of PD pixels and a plurality of micro lenses covering the PD pixels is shown.

[0034] Figure 3B A group of four pixels of one color is shown.

[0035] Figure 4 The Bayer pattern consisting of each individual pixel of the PD image sensor is shown.

[0036] Figure 5 A full resolution interpolated green image is shown with no missing green pixels.

[0037] Fig. 6A A defocused portion of the interpolated green image is shown.

[0038] Figure 6B An in-focus portion of the full-resolution interpolated green image is shown.

[0039] Figure 7 This is the re-mosaicing method according to the present invention.

[0040] Figure 8 The joint image filtering process according to the present invention is shown.

[0041] Fig. 9 The joint least squares fitting process according to the present invention is shown.

[0042] Corresponding reference numerals indicate corresponding components throughout the several views of the drawings. It should be appreciated by those skilled in the art that the elements in the drawings are illustrated for simplicity and clarity only and are not necessarily drawn to scale. For example, the dimensions of some elements in the drawings may be exaggerated relative to other elements to help improve understanding of various embodiments of the present invention.

[0043] Description of Figure Numbers

[0044] 100, 120, 300: pixel array;

[0045] 102, 114, 122, 322: microlenses;

[0046] 104: left phase detection pixel;

[0047] 106: right phase detection pixel;

[0048] 108, 110: light;

[0049] 112: image pixels;

[0050] 124: upper left PD pixel;

[0051] 126: upper right PD pixel;

[0052] 128: lower left PD pixel;

[0053] 130: lower right PD pixel;

[0054] 202: left image;

[0055] 204: right image;

[0056] 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316: PD pixels;

[0057] 324, 326, 328, 330, 344: pixels;

[0058] 340, 400: Bayer type;

[0059] 342: Super Pixel;

[0060] 500: green image;

[0061] 700: Re-mosaic method;

[0062] 702, 704, 706, 708, 710, 712, 714, 716, 718, 720, 722: frame;

[0063] 800: Joint image filtering process;

[0064] 802, 902: full resolution interpolated green image;

[0065] 804, 904: blurred image;

[0066] 806: point P(x,y);

[0067] 808: Point Q(x,y);

[0068] 810: Space core;

[0069] 812: Bilateral kernel;

[0070] 814, 908: corrected full-resolution interpolated green image;

[0071] 816: In-range nuclear;

[0072] 900: Joint least squares fitting process;

[0073] 906: Window. DETAILED DESCRIPTION

[0074] In the following description, many specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that it is not necessary to adopt the specific details to practice the present invention. In other examples, well-known materials or methods are not described in detail in order to avoid confusing the present invention.

[0075] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearance of the phrase "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in one or more embodiments in any suitable combinations and / or subcombinations.

[0076] Figure 1A A pixel array 100 of a phase detection (PD) image sensor with PDAF capability is shown. In an embodiment, two pixels, such as a left phase detection (PD) pixel and a right PD pixel, are covered by a microlens. The phase difference between the left PD pixel and the right PD pixel is detected to perform autofocus. The pixel array 100 includes a plurality of PD pixels, and a microlens 102 covers a left PD pixel 104 and a right PD pixel 106. The PD pixel 104 and the PD pixel 106 form a PD pixel unit. The pixel array 100 may also include a plurality of image pixels 112. Each image pixel has a microlens 114. The image pixel may not be used as a PD pixel, but the PD pixel may be used as an image pixel.

[0077] Figure 1BA cross section of a microlens 102 covering a left PD pixel 104 and a right PD pixel 106 is shown. Light 108 from the left side in a camera (not shown) is incident at the microlens 102 and is directed to the right PD pixel 106. Light 110 from the right side in a camera (not shown) is incident at the microlens 102 and is directed to the left PD pixel 104. The left PD pixel 104 detects a left image of a PD image as an image detected by a PD image sensor. The right PD pixel 106 detects a right image of the PD image. The PD pixel 104 and the PD pixel 106 may be identified as a first left PD pixel and a first right PD pixel, respectively.

[0078] In an embodiment, four pixels are covered by a microlens. Figure 1C As shown in . Figure 1C Pixel array 120 including a plurality of PD pixels and a plurality of microlenses covering the PD pixels is shown. Microlens 122 covers upper left PD pixel 124, upper right PD pixel 126, lower left PD pixel 128, and lower right PD pixel 130. PD pixel 124, PD pixel 126, PD pixel 128, and PD pixel 130 form a PD pixel unit. Pixel array 120 may also include a plurality of image pixels 112. Each image pixel has a microlens 114. An image pixel may not be used as a PD pixel, but a PD pixel may be used as an image pixel.

[0079] A one-dimensional cross-section of the microlens 122 covering the PD pixel 124, the PD pixel 126, the PD pixel 128, and the PD pixel 130 is Figure 1D As shown in . Figure 1D Similar to Figure 1B , wherein microlens 102 is replaced by microlens 122, PD pixel 104 is replaced by PD pixel 124, and PD pixel 106 is replaced by PD pixel 126. PD pixel 128 and PD pixel 130 are not shown in the cross section. PD pixel 124, PD pixel 126, PD pixel 128, and PD pixel 130 may be identified as a first left PD pixel, a first right PD pixel, a second left PD pixel, and a second right PD pixel, respectively.

[0080] Figure 2A to Figure 2C A left image of a PD image and a right image of a PD image formed by left and right PD pixels, respectively, are shown. A PD image refers to an image detected by a PD image sensor before any processing. A PD image has a left image and a right image. Figure 2A The left image 202 is shown separated to the left from the right image 204 when the PD image is defocused (eg, front defocused). Figure 2B A left image 202 is shown overlapping a right image 204 when the images are in focus. Figure 2CThe left image 202 is shown separated to the right from the right image 204 when the PD image is defocused (eg, post-defocused). The imaging lens of the camera (not shown) is moved to achieve Figure 2B , namely, left image 202 overlapped with right image 204. However, since not all objects captured in a PD image are at the same distance from the camera, some portions of the PD image may be in focus and some portions of the PD image may be out of focus.

[0081] In an embodiment, all pixels of the pixel array may be PD pixels. In an embodiment, every four pixels may be covered by a microlens. Figure 3A and Figure 3B As shown in . Figure 3A Pixel array 300 including a plurality of PD pixels and a plurality of micro lenses covering the PD pixels are shown. Micro lens 322 covers four PD pixels, such as pixel 324, pixel 326, pixel 328, and pixel 330. For example, PD pixel 301, PD pixel 302, PD pixel 303, and PD pixel 304 may be blue pixels. PD pixel 305, PD pixel 306, PD pixel 307, and PD pixel 308 may be green pixels. PD pixel 309, PD pixel 310, PD pixel 311, and PD pixel 312 may also be green pixels. PD pixel 313, PD pixel 314, PD pixel 315, and PD pixel 316 may be red pixels. Figure 3B Pixel array 300 is shown to better illustrate the distribution of color pixels. It should be appreciated that other distributions of color pixels are possible.

[0082] exist Figure 3A In the embodiment shown in , pixel 324, pixel 326, pixel 328, and pixel 330 are used not only as PD pixels but also as image pixels. Figure 3B All pixels in the pixel array 300 shown in FIG. 3 serve as PD pixels as well as image pixels. When the phase difference between a pair of pixels covered by the same microlens is determined, the pixel serves as a PD pixel. When the pixel is read individually to form a PD image, it serves as an image pixel.

[0083] Figure 3B A group of four pixels is shown as one color. In addition, Figure 3B A Bayer pattern 340 with 2×2 super pixels is shown. Each super pixel, such as super pixel 342, is composed of 2×2 pixels (e.g., pixel 344) of the PD image sensor (which is a PD pixel as well as an image pixel). In order to enhance the resolution of the PD image detected by the PD image sensor, the pixels contained in the super pixels are redistributed to form a Bayer pattern 400 composed of each individual pixel of the PD image sensor, Figure 4As shown in . It is worth noting that the Bayer pattern 400 is not a physical pixel array. In contrast, the pixel array 300 is a physical pixel array. This process is called re-mosaicing the Bayer image. The re-mosaiced Bayer image of the Bayer pattern 400 has a green image, a blue image, and a red image. Figure 3B Compared to the Bayer pattern 340 image with super pixels, the re-mosaiced Bayer image has double the resolution in the x-direction and the y-direction.

[0084] When re-mosaicing a Bayer image, the green image is first interpolated, e.g. Figure 3B The Bayer pattern 340 is used to obtain a full-resolution interpolated green image. The full-resolution interpolated green image is referred to as a green image 500 without missing green pixels. Figure 5 As shown in .

[0085] By including FIG. 3A to FIG. 3B The PD image captured by the PD image sensor of the pixel array 300 may have some portions that are in focus and some portions that are out of focus. Figure 1D , in the focused portion, the left PD pixel 124 and the right PD pixel 126 of the PD pixel unit detect the same light intensity because the left image 202 of the PD image overlaps with the right image 204 of the PD image. Figure 2B As shown in .

[0086] However, in the defocused portion, the left PD pixel 124 and the right PD pixel 126 detect different light intensities because the left image 202 of the PD image is offset from the right image 204 of the PD image. Figure 2A or Figure 2C As shown in . The left pixel of a PD pixel unit is adjacent to the right pixel of the same PD pixel unit. Therefore, in the defocused portion, two adjacent pixels (e.g., the left PD pixel and the right PD pixel) have different intensities. This produces artifacts in the captured PD image.

[0087] Interpolation algorithms for red, green, and blue images are available. They are not discussed in detail here, as they are not important to the present invention. In the present disclosure, it is understood that a full-resolution interpolated green image is obtained or has been obtained. The full-resolution interpolated green image contains defocused portions with artifacts and focused portions with sharp images. The full-resolution interpolated green image is then corrected using the disclosed method of the present invention by removing artifacts but maintaining a sharp image. The corrected full-resolution interpolated green image is then used to re-mosaic the Bayer image with the artifacts removed and the sharp image unaffected.

[0088] Fig. 6A shows the defocused portion of the full resolution interpolated green image. In this example, the entire Fig. 6Ais a defocused image. The defocused portion shows an artifact in the image. Simply put, the artifact appears as a white pixel between two black pixels. For example, the white pixel is the left PD pixel of a PD pixel unit, and the black pixel is the right PD pixel of the same PD pixel unit and the right PD pixel of an adjacent PD pixel unit. For comparison, Figure 6B shows an in-focus portion of the full-resolution interpolated green image, which shows a sharp image. In this example, the entire Figure 6B to focus the image. Figure 6B As shown in , no artifacts appear in the focused portion.

[0089] Fig. 6A The artifacts in the defocused portion shown in the figure can be easily removed using a low pass filter. However, the low pass filter will Figure 6B The sharp image in the focused portion shown in the figure is blurred while removing artifacts.

[0090] According to the present invention, in order to remove artifacts in the defocused part but keep a clear image in the focused part, Figure 7 As shown in FIG. 7 , a re-mosaicing method 700 is disclosed. Block 702 indicates that, for example, Figure 3B The PD image sensor of the pixel array 300 shown in FIG. 704 shows an optional step as sensitivity correction.

[0091] Return to reference Figure 1D , ideally, light 108 from the left side is directed to the right PD pixel 126, and light 110 from the right side is directed to the left PD pixel 124. However, light 108 may be partially deflected to the left PD pixel 124, and light 110 may be partially deflected to the right PD pixel 126. In order to minimize the effect of light 108 on the left PD pixel 104 and light 110 on the right PD pixel 106, sensitivity correction 704 may be performed. The sensitivity of the green image, the blue image, and the red image may be corrected. Block 704 may generate the sensitivity-corrected green image, the blue image, and the red image.

[0092] Box 706 shows the process of obtaining a full-resolution interpolated green image from the PD image of box 702. Various algorithms are available. Here, they are not discussed in detail because they are not important to the present invention. Box 708 shows the full-resolution interpolated green image after the full-resolution green interpolation of box 706 is completed. The full-resolution interpolated green image contains a portion with artifacts and a portion with a clear image. Box 710 shows a low-pass filtering process.

[0093] Box 712 shows the blurred image after the low pass filtering of box 710 is completed. The artifacts of the full resolution interpolated green image of box 708 have been removed, but the sharp image in the full resolution interpolated green image of box 708 has become blurred.

[0094] Block 714 shows a joint image filtering process or a joint least squares fitting process. The joint image filtering and / or joint least squares fitting jointly involves the full resolution interpolated green image of block 708 and the blurred image of block 712. The full resolution interpolated green image of block 706 and the blurred image of block 712 are combined or joined in this process.

[0095] Box 716 shows the corrected full-resolution interpolated green image after the joint image filtering or joint least squares fitting process of box 714. Artifacts in the full-resolution interpolated green image of box 708 have been removed, and the clear image in the full-resolution interpolated green image of box 708 is not affected. Box 718 shows an optional step of sharpening the green image. In this box, the corrected full-resolution interpolated green image of box 716 can be further sharpened. Various sharpening algorithms including high pass filtering are available.

[0096] Block 720 shows the interpolation of red and blue based in part on the corrected full resolution interpolated green image of block 716 or its sharpened image of block 718. It should be appreciated that the interpolation of block 720 is not used to produce full resolution interpolated red and blue images. Instead, it is used to produce Figure 4 4. The Bayer image shown in FIG. 400 of the Bayer pattern 400 of FIG. 400 includes a green image with missing green pixels (two pixels are missing in the Bayer pattern), a blue image with missing blue pixels (three pixels are missing in the Bayer pattern), and a red image with missing red pixels (three pixels are missing in the Bayer pattern). In contrast, the full resolution interpolated image has no missing pixels. Block 722 shows the re-mosaiced Bayer image after the re-mosaicing method 700. The re-mosaiced Bayer image is formed using the interpolated blue and red images of block 720 and a portion of the corrected interpolated green image of block 716.

[0097] In block 720, if block 704 is skipped, the red and blue images are interpolated based in part on the corrected full-resolution interpolated green image of block 716 or its sharpened image of block 718 and in part on the color image arranged in the original Bayer pattern, which is the PD image shown in block 702. If block 704 is not skipped, the red and blue images are interpolated based in part on the sensitivity-corrected green, blue, and red images produced at block 704 instead of the PD image of block 702.

[0098] The re-mosaic method 700 may be executed by a processor of an imaging system. The imaging system includes a PD image sensor, the PD image sensor including a plurality of PD pixel units, each PD pixel unit including a first left PD pixel and a first right PD pixel, the first left PD pixel and the first right PD pixel being covered by a microlens. The plurality of PD pixel units include a green PD pixel unit forming a green image, a blue PD pixel unit forming a blue image, and a red PD pixel unit forming a red image. The green image, the blue image, and the red image form a color image arranged in an original Bayer pattern, the original Bayer pattern including a green PD pixel unit, a blue PD pixel unit, and a red PD pixel unit, the color image being the PD image shown in frame 702.

[0099] The re-mosaic method 700 may be executed by a processor of an imaging system. The imaging system comprises: a plurality of green pixel units, wherein a microlens covers each unit and each unit has four pixels, Figure 3A and Figure 3B ; a plurality of blue pixel cells, wherein a microlens covers each cell and each cell has four pixels; and a plurality of red pixel cells, wherein a microlens covers each cell and each cell has four pixels. A plurality of green pixel cells form a green image, a plurality of blue pixel cells form a blue image, and a plurality of red pixel cells form a red image. The green image, the blue image, and the red image form a color image arranged in an original Bayer pattern, the original Bayer pattern including green pixel cells, blue pixel cells, and red pixel cells, the color image being an image similar to the PD image shown in box 702. In this embodiment, the pixel cell is not a PD pixel cell, which means that the phase difference between the two pixels is not detected.

[0100] In an embodiment, the microlens covers a pixel unit, and the pixel unit may have 2×2, 4×4, 8×8 or 16×16 pixels. In principle, the pixel unit may have m×m pixels, where m is any integer.

[0101] Figure 8 It shows that according to the present invention Figure 7 The joint image filtering process 800 shown in block 714 of FIG. Figure 7 The full resolution interpolated green image 802 shown in box 708 in FIG. 8 is the target image, which is spatially variant convolved with the bilateral kernel 812 to produce Figure 7 The corrected full resolution interpolated green image 814 is shown in box 716 in FIG. The bilateral kernel 812 is a joint bilateral function.

[0102] Also there Figure 7The blurred image 804 shown in box 712 in FIG. 8 is used to generate a range kernel 816. For example, the blurred image 804 may be a low pass filtered image of the full resolution interpolated green image 802. The range kernel 816 is typically a window that is much smaller than the entire blurred image 804. For example, the range kernel 816 may be a 3×3 or 5×5 window. The range kernel 816 varies with the blurred image 804. When the local variation of the blurred image 804 is small, the range kernel 816 is close to one, and when the local variation of the blurred image 804 is large, the range kernel 816 is close to zero.

[0103] For example, the 3×3 kernel 816 at point Q(x, y) 808 of the blurred image 804 can be expressed as

[0104]

[0105] Where i = -1, 0, 1 and j = -1, 0, 1. R(x, y) is the range kernel 816 at Q(x, y) 808 of the blurred image 804. I(x, y) is the blurred image 804 at Q(x, y) 808, and I(x+iΔx, y+jΔy) is the blurred image 804 at the sampling points around Q(x, y) 808 in the window. Δx and Δy are predetermined constants indicating the sample interval. σ is a constant.

[0106] When the local variation of the blurred image 804 (i.e., the square of the absolute value of I(x,y)-I(x+iΔx,y+jΔy)) is small, R(x,y)i,j is close to one, and when the local variation of the blurred image 804 (i.e., the square of the absolute value of I(x,y)-I(x+iΔx,y+jΔy)) is large, the in-range kernel 816 is close to zero.

[0107] In this example, the 3×3 range kernel 816 at Q(x,y) 808 of the blurred image 804 can be expressed as

[0108]

[0109] The in-range kernel 816 is multiplied by the spatial kernel 810 (e.g., a Gaussian function) to obtain the bilateral kernel 812 (e.g., a truncated Gaussian function at a point). Both the spatial kernel 810 and the bilateral kernel 812 can be the same window as the in-range kernel 816, for example 3×3. The bilateral kernel 812 at point P(x,y) 806 of the full-resolution interpolated green image 802 is the product of the spatial kernel 810 and the in-range kernel 816. The center of the spatial kernel 810 is aligned with the center of the in-range kernel 816, and the in-range kernel corresponds to point Q(x,y) 808 of the blurred image 804. Point Q(x,y) 808 of the blurred image 804 is aligned with point P(x,y) 806 of the full-resolution interpolated green image 802, at Figure 8 As shown in .

[0110] In the spatially-variant convolution between the full-resolution interpolated green image 802 and the bilateral kernel 812, the kernel (i.e., the bilateral kernel 812) changes during the convolution as the point P(x,y) 806 changes or moves. In this way, the sharp image (the in-focus portion of the interpolated green image) will not be smoothed by the bilateral kernel 812, while the defocused portion of the interpolated green image having artifacts is smoothed by the bilateral kernel 812.

[0111] The range kernel 816, the spatial kernel 810, and the bilateral kernel 812 are not limited to 3×3 or 5×5 windows. They include m×n windows, where m and n can be any numbers. Preferably, m and n are odd integers.

[0112] Fig. 9 The joint least squares fitting process 900 according to the present invention is shown. Figure 7 As shown in block 714 in FIG. 1 , a joint least squares fit of the full-resolution interpolated green image and the blurred image is performed within the scan window to obtain a corrected full-resolution interpolated green image. Figure 7 The full resolution interpolated green image 902 shown in box 708 in FIG. 1 is the target image. Figure 7 The blurred image 904 shown in box 712 in is the guide image. For example, the blurred image 904 may be a low pass filtered image of the full resolution interpolated green image 902. Window 906 covers the target image (full resolution interpolated green image 902) and the guide image (blurred image 904). For example, window 906 includes 3×3 points of the full resolution interpolated green image 902, which have values ​​of T1, T2, T3, T4, T5, T6, T7, T8, and T9, respectively. T represents the target. Window 906 also includes 3×3 points of the blurred image 904, which have values ​​of G1, G2, G3, G4, G5, G6, G7, G8, and G9, respectively. G represents the guide. A joint least squares fitting process is performed to determine a and b, which will result in the minimum value of the square of the following

[0113]

[0114] The above expression can be written as

[0115] a Gi+b-Ti,

[0116] Wherein Gi is the value of the i-th point of the blurred image 904 in the 3×3 window 906, and i is 1, 2, 3, ..., 9, and Ti is the value of the i-th point of the full-resolution interpolated green image 902 in the 3×3 window 906, and i is 1, 2, 3, ..., 9.

[0117] After determining a and b, the window 906 is also Figure 7 The corrected full resolution interpolated green image 908 shown in box 716 in FIG. 1 is provided by

[0118] Cj=a Gj+b,

[0119] Where j indicates the center point of the 3×3 window 906 covering the point with i (which is 1, 2, 3, . . . , 9), where Cj is the value of the center point of the 3×3 window 906 at the corrected full-resolution interpolated green image 908, and where Gj is the value of the center point of the 3×3 window 906 at the blurred image 904. In this example, j is 5. The values ​​of all points of the corrected full-resolution interpolated green image 908 can be obtained by scanning the window 906.

[0120] Window 906 is not limited to containing 3×3 points. It can cover m×n points of the full-resolution interpolated green image 902 and the blurred image 904, where m and n can be any numbers. Preferably, m and n are odd integers. In this way, the sharp image (the focused portion of the full-resolution interpolated green image) will not be smoothed by the joint least squares fitting process, while the defocused portion of the full-resolution interpolated green image with artifacts is smoothed by the joint least squares fitting process.

[0121] While the invention has been described herein with respect to exemplary embodiments and the best mode for practicing the invention, it will be apparent to those skilled in the art that many modifications, improvements, and sub-combinations of various embodiments, adaptations, and variations may be made to the invention without departing from the spirit and scope of the invention.

[0122] The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification and the claims. Rather, the scope should be determined entirely by the appended claims, which should be interpreted in accordance with established doctrines of claim interpretation. The specification and drawings should, therefore, be regarded as illustrative rather than restrictive.

Claims

1. An imaging system, characterized in that: include: A phase detection image sensor, comprising a plurality of phase detection pixel units, each phase detection pixel unit comprising a first left phase detection pixel and a first right phase detection pixel, wherein the first left phase detection pixel and the first right phase detection pixel are covered by a microlens; wherein the plurality of phase detection pixel units include a green phase detection pixel unit for forming a green image, a blue phase detection pixel unit for forming a blue image, and a red phase detection pixel unit for forming a red image; as well as wherein the green image, the blue image and the red image form a color image arranged in an original Bayer pattern, the original Bayer pattern including the green phase detection pixel unit, the blue phase detection pixel unit and the red phase detection pixel unit; Processor, configured as: interpolating the green image to obtain a full-resolution interpolated green image including a defocused portion with artifacts and a focused portion with a sharp image; performing low pass filtering on the full resolution interpolated green image to obtain a blurred image of the full resolution interpolated green image; combining the full-resolution interpolated green image and the blurred image of the full-resolution interpolated green image to obtain a corrected full-resolution interpolated green image; The artifacts of the defocused portion of the full-resolution interpolated green image are removed, and the sharp image of the focused portion of the full-resolution interpolated green image is not affected.

2. The imaging system according to claim 1, characterized in that Each phase detection pixel unit further includes a second left phase detection pixel and a second right phase detection pixel, and the first left phase detection pixel, the first right phase detection pixel, the second left phase detection pixel and the second right phase detection pixel are covered by a microlens.

3. The imaging system according to claim 1, characterized in that The processor is further configured to: The full-resolution interpolated green image is convolved spatially with a bilateral kernel, wherein the bilateral kernel is the product of a spatial kernel and an in-range kernel.

4. The imaging system according to claim 3, characterized in that The spatial kernel comprises a Gaussian function.

5. The imaging system according to claim 3, characterized in that: The in-range kernel is a function of the blurred image, and when the local variation of the blurred image is small, the in-range kernel approaches one, and when the local variation of the blurred image is large, the in-range kernel approaches zero.

6. The imaging system according to claim 1, characterized in that: The processor is further configured to: A joint least squares fitting of the full-resolution interpolated green image and the blurred image is performed within a scanning window of m×n points including the full-resolution interpolated green image and the blurred image to obtain a corrected full-resolution interpolated green image, where m and n are odd integers.

7. The imaging system according to claim 6, characterized in that: The joint least squares fitting includes: determining a and b to obtain the minimum value of the square of (aGi+b-Ti) within the scanning window, where Gi is the value of the blurred image and Ti is the value of the full-resolution interpolated green image; and using the determined a and b to determine the corrected full-resolution interpolated green image as Cj=aGj+b, where j indicates the center point of the scanning window, and where Cj is the value of the corrected full-resolution interpolated green image at the scanning window, and Gj is the value of the blurred image at the scanning window.

8. The imaging system according to claim 1, characterized in that: The processor is further configured to: The sensitivity of the green image, the blue image, and the red image are corrected before interpolating the green image.

9. The imaging system according to claim 1, characterized in that: The processor is further configured to: After obtaining the corrected full-resolution interpolated green image, the corrected full-resolution interpolated green image is sharpened.

10. The imaging system according to claim 1, characterized in that The processor is further configured to: interpolating the blue image and the red image based in part on the corrected full resolution interpolated green image and in part on the color image arranged in the original Bayer pattern; as well as A Bayer image is formed using the interpolated blue image and the interpolated red image and portions of the corrected full-resolution interpolated green image.

11. A method for re-mosaicing a Bayer image from a color image arranged in an original Bayer pattern, characterized in that The original Bayer pattern includes a green pixel unit, a blue pixel unit and a red pixel unit, and the method includes: interpolating the green image to obtain a full-resolution interpolated green image including a defocused portion having artifacts and a focused portion having a sharp image; performing low pass filtering on the full resolution interpolated green image to obtain a blurred image of the full resolution interpolated green image; combining the full-resolution interpolated green image and the blurred image of the full-resolution interpolated green image to obtain a corrected full-resolution interpolated green image; The artifacts of the defocused portion of the full-resolution interpolated green image are removed, and the sharp image of the focused portion of the full-resolution interpolated green image is not affected.

12. The method for re-mosaicing a Bayer image from a color image arranged in an original Bayer pattern according to claim 11, characterized in that Also includes: A spatially-variant convolution of the full-resolution interpolated green image with a bilateral kernel is performed, wherein the bilateral kernel is a product of a spatial kernel and a range kernel.

13. The method for re-mosaicing a Bayer image from a color image arranged in an original Bayer pattern according to claim 12, characterized in that The spatial kernel comprises a Gaussian function.

14. The method for re-mosaicing a Bayer image from a color image arranged in an original Bayer pattern according to claim 12, characterized in that The in-range kernel is a function of the blurred image, and when the local variation of the blurred image is small, the in-range kernel approaches one, and when the local variation of the blurred image is large, the in-range kernel approaches zero.

15. The method for re-mosaicing a Bayer image from a color image arranged in an original Bayer pattern according to claim 11, characterized in that Also includes: A joint least squares fitting of the full-resolution interpolated green image and the blurred image is performed within a scanning window of m×n points including the full-resolution interpolated green image and the blurred image to obtain a corrected full-resolution interpolated green image, where m and n are odd integers.

16. The method for re-mosaicing a Bayer image from a color image arranged in an original Bayer pattern according to claim 15, characterized in that The joint least squares fitting includes: determining a and b to obtain the minimum value of the square of (aGi+b-Ti) within the scanning window, where Gi is the value of the blurred image and Ti is the value of the full-resolution interpolated green image; and using the determined a and b to determine the corrected full-resolution interpolated green image as Cj=aGj+b, where j indicates the center point of the scanning window, and where Cj is the value of the corrected full-resolution interpolated green image at the scanning window, and Gj is the value of the blurred image at the scanning window.

17. The method for re-mosaicing a Bayer image from a color image arranged in an original Bayer pattern according to claim 11, characterized in that Also includes: The sensitivity of the green image, the blue image, and the red image are corrected before interpolating the green image.

18. The method for re-mosaicing a Bayer image from a color image arranged in an original Bayer pattern according to claim 11, characterized in that Also includes: After obtaining the corrected full-resolution interpolated green image, the corrected full-resolution interpolated green image is sharpened.

19. The method for re-mosaicing a Bayer image from a color image arranged in an original Bayer pattern according to claim 11, characterized in that Also includes: interpolating a blue image and a red image based in part on the corrected full resolution interpolated green image and in part on the color image arranged in the original Bayer pattern; as well as A re-mosaiced Bayer image is formed using the interpolated blue image and the interpolated red image and portions of the corrected full-resolution interpolated green image.

20. An imaging system, characterized in that: include: A plurality of green pixel units, each unit is covered by a microlens and each unit has four pixels; A plurality of blue pixel units, each unit having four pixels and each unit covered by a microlens; A plurality of red pixel units, each unit is covered by a microlens and each unit has four pixels; wherein the plurality of green pixel units form a green image, the plurality of blue pixel units form a blue image, and the plurality of red pixel units form a red image; and wherein the green image, the blue image and the red image form a color image arranged in an original Bayer pattern, the original Bayer pattern including the green pixel unit, the blue pixel unit and the red pixel unit; Processor, configured as: interpolating the green image to obtain a full-resolution interpolated green image including a defocused portion with artifacts and a focused portion with a sharp image; performing low pass filtering on the full resolution interpolated green image to obtain a blurred image of the full resolution interpolated green image; combining the full-resolution interpolated green image and the blurred image of the full-resolution interpolated green image to obtain a corrected full-resolution interpolated green image; The artifacts of the defocused portion of the full-resolution interpolated green image are removed, and the sharp image of the focused portion of the full-resolution interpolated green image is not affected.

21. The imaging system according to claim 20, characterized in that The processor is further configured to: The full-resolution interpolated green image is convolved spatially with a bilateral kernel, wherein the bilateral kernel is the product of a spatial kernel and an in-range kernel.

22. The imaging system according to claim 20, characterized in that The processor is further configured to: A joint least squares fitting of the full-resolution interpolated green image and the blurred image is performed within a scanning window of m×n points including the full-resolution interpolated green image and the blurred image to obtain a corrected full-resolution interpolated green image, where m and n are odd integers.

Citation Information

Patent Citations

  • Image demosaicer and method

    CN110365961A

  • Moving Robot and controlling method

    KR1020210004677A

  • Efficient Image Demosaicing and Local Contrast Enhancement

    US20150042775A1

  • Imaging apparatus and image processing method of thereof

    US20170163951A1

  • Dual phase detection power optimizations

    US20190335089A1