Image demosaicing circuits for the reduction of artifacts

Abstract

Systems and circuits for demosaicing images with minimal artifacts are disclosed. The circuits are designed to calculate a color value within a first color space for the first pixel of a plurality of pixels, based at least on a color value within the first color space of a first group of pixels adjacent to the first pixel, and to calculate a first chrominance value within a second color space for the first pixel, based at least on the calculated color value within the first color space. The circuits are also designed to calculate a luminance value within the second color space for the first pixel, based at least on the first chrominance value within the second color space, and to convert the luminance value within the second color space and the first chrominance value within the second color space into an output pixel value within a third color space.

Description

Technical area

[0001] At least one embodiment relates to circuits for demosaicing color samples (color sampling values ​​or color measurements) to reconstruct a full-color image with minimal artifacts. background

[0002] Demosaicing is a digital imaging technique used to reconstruct a full-color image from incomplete color samples output by an image sensor overlaid with a color filter array (CFA), such as a Bayer filter. A CFA is common in digital cameras and other imaging devices, allowing a single sensor to capture color information by filtering the light hitting each pixel according to a specific color (e.g., red, green, or blue). Because of this process, each pixel captures only one of the three primary colors, so the missing colors must be interpolated for each pixel. The goal of demosaicing is to reconstruct a high-quality full-color image by estimating the colors not captured by each pixel.Various algorithms are used to derive the missing information based on the color information of neighboring pixels. Brief description of the drawings Fig. Figure 1 is a block diagram of an exemplary system for performing image demosaicing with minimal artifacts, according to at least one embodiment; Fig. Figure 2 is a block diagram of an exemplary data flow for demosaicing color samples to obtain a full-color output, according to at least one embodiment; Fig. 3A is a block diagram of an exemplary kernel of a multiphase filter according to at least one embodiment; Fig. 3B is a block diagram of an exemplary kernel of a multiphase filter according to at least one embodiment; Fig. 3C is a block diagram of an exemplary kernel of a multiphase filter according to at least one embodiment; Fig. 3D is a block diagram of an exemplary kernel of a multiphase filter according to at least one embodiment; Fig. Figure 4 is a block diagram of an exemplary data flow for a luma direction improvement according to at least one embodiment; Fig. Figure 5 is a flowchart of an exemplary method for image demosaicing with minimal artifacts according to at least one embodiment; and Fig. Figure 6 is a block diagram illustrating an exemplary computer system according to at least one embodiment of the present disclosure. Detailed description

[0003] While demosaicing an image is essential for creating full-color digital images from raw sensor data, it can introduce various side effects. The process can cause different types of artifacts, such as moiré patterns, incorrect colors, and / or zippering at high-contrast edges. Interpolation during demosaicing can sometimes blur fine details, especially in areas with subtle textures or high-frequency details. Furthermore, achieving accurate color reproduction can be challenging, and demosaicing can sometimes result in color shifts, where the colors in the final image do not precisely match the original scene.

[0004] The present disclosure provides systems and methods that enable the reconstruction of a full-color image with minimal artifacts by demosaicing color samples. Color samples (e.g., a color filter array (CFA)) can be demosaiced to reconstruct a full-color image corresponding to the color samples. The color samples can be in a first (e.g., input) color space (e.g., RGB) and be converted to a second (e.g., output) color space (e.g., YUV). In some embodiments, the color samples are converted to an intermediate color space (e.g., YUViVi) before being converted to the second (e.g., output) color space. In the first color space, each pixel can have a single value and need not have a value for every component of the color space. For example, a first pixel can have only an R value, without a G or B value.A second pixel can only have a G value, without an R or B value, and so on. In the second color space, each pixel can have a value for each component of the output color space. For example, each pixel can have a luminance value (luma) (e.g., Y), a blue chrominance value (chroma) (e.g., U, Cb), and a red chrominance value (chroma) (e.g., V, Cr). The values ​​of the first color space can be converted to the values ​​of the second color space using chroma-guided interpolation and luma estimation. In some embodiments, the chroma-guided interpolation includes chroma filtering and / or chroma filtering with trimming. In some embodiments, additional processing is performed during the demosaicing process, such as a luma zipper (LZF) filter and / or luma direction enhancement (LDE).

[0005] During chroma-guided interpolation, the pixel values ​​of color samples in the first color space (e.g., RGB values) can be converted to values ​​in an intermediate color space (e.g., YiUiVi values). The color samples can contain a CFA with one or more pixels. Each pixel can have a single value corresponding to a component of the first color space. In some embodiments, the CFA is obtained using an RGGB Bayer filter. In such a case, since there are more G values ​​than R or B values, upsampling filters and / or interpolation can be performed to determine a G value for each pixel of the CFA. Based on these G values, values ​​within the intermediate color space can be calculated. For example, first chroma values ​​(e.g., Ui values) can be calculated on pixels of the CFA that have a B value, and second chroma values ​​(e.g., Vi values) can be calculated on pixels of the CFA that have an R value.Upsampling and / or interpolation can be performed again to determine intermediate chroma values ​​(e.g., Ui and Vi values) for each pixel of the CFA. In some embodiments, the intermediate chroma values ​​Ui and Vi can be smoothed using a low-pass filter to obtain smoothed intermediate chroma values ​​Uc and Vc.

[0006] In some embodiments, the intermediate luminance values ​​(e.g., Yi values) can be calculated using the following formula: Yi = 0.25*R + 0.5*G + 0.25*B. In some embodiments, the intermediate chroma values ​​(e.g., Ui values ​​and Vi values) can be calculated using the following formulas: Ui = 0*R - 0.5*G + 0.5*B and Vi = 0.5*R - 0.5*G + 0*B.

[0007] During luma estimation, intermediate luma values ​​(e.g., Yi values), corresponding to the luminance component of the intermediate color space (e.g., YiUiVi format), can be calculated for each pixel of the CFA based on the intermediate chroma values ​​(e.g., Ui and Vi values) calculated during chroma-guided interpolation and the color samples of each pixel. For example, the color sample value of a given pixel can be combined with the calculated Ui and Vi values ​​for that pixel to obtain the Yi value for the pixel. In some implementations, different weights are applied to the Ui and Vi values ​​(e.g., a chroma correction) based on the pixel's color in the CFA. For example, the Yi value for an R pixel can be calculated using the following formula: Yi = R + 0.5Ui - 1.5Vi. The Yi value for a G-pixel can be calculated using the following formula: Yi = G + 0.5Ui + 0.5Vi.The Yi value for a B-pixel can be calculated using the following formula: Yi = B - 1.5Ui + 0.5Vi. In some embodiments, the smoothed values ​​(e.g., Uc, Vc) are used instead of the intermediate values ​​(e.g., Ui, Vi) to obtain a luma estimate (e.g., Yc).

[0008] In some embodiments, a luma zipper filter can be applied to the intermediate luma values ​​(e.g., Yi values, Yc values) to remove (or reduce) zipper artifacts that would appear in the full-color image. The luma zipper filter can be a bilateral low-pass filter. In some embodiments, the luma zipper filter is a two-dimensional filter, so that the output is a linear combination of a window of input intermediate luma values ​​weighted by fixed filter kernel coefficients and dynamic range kernel coefficients that depend on the input range and the input intermediate luma values. The fixed filter kernel can have a wide passband to preserve the high-frequency content (e.g., textures, image details, etc.) in the full-color image. In some embodiments, the improved value of a single intermediate luma value can be calculated based on a 9x9 patch of input intermediate luma values.

[0009] In some embodiments, a direction filter can be applied to the intermediate luma values ​​(e.g., Yi values, Yc values) to smooth edges in the final full-color image. The direction filter can start with a 9x9 patch of raw values ​​(e.g., from the CFA), apply a bilinear LPF to calculate a 7x7 patch of smoothed luma values, detect edge directions within a 5x5 patch of the smoothed luma values, and then modify the intermediate luma value centered in the 5x5 patch based on the detected direction.

[0010] The resulting intermediate values ​​(e.g., YiUiVi, YcUcVc) can be converted to the output color space (e.g., YUV) using one or more operations. For example, the intermediate values ​​can be multiplied by one or more transformation matrices to obtain the output values.

[0011] The advantages of the disclosed methods include, among others, improved image quality and reduced visual artifacts after converting raw CFA color sample values ​​into a full-color image.

[0012] Fig. Figure 1 is a block diagram of an exemplary image demosaicing system 102 for performing image demosaicing with minimal artifacts according to at least one embodiment. In some embodiments, the image demosaicing system 102 may be included as part of a system-on-a-chip (SoC). In some embodiments, an image demosaicing circuit 106 may be included as part of an SoC. In some embodiments, the image demosaicing system 102, the image demosaicing circuit 106 and / or one or more circuits of the image demosaicing circuit 106 (e.g. a raw interpolation circuit 108, a chroma estimation circuit 110, a luma estimation circuit 112, a color space transformation circuit 114, a luma zipper filter circuit 116, a luma direction enhancement circuit 118) may be included as part of an image processing pipeline.

[0013] The image demosaicing system 102 can include a memory 104 and an image demosaicing circuit 106. The memory 104 can contain one or more registers, one or more caches (e.g., L1 cache, L2 cache, etc.), and / or main memory (e.g., RAM, DRAM, SRAM, SDRAM, DDR SDRAM, etc.). The memory 104 can store information related to an image demosaicing process, such as raw pixel values, color sample values, intermediate pixel values, output pixel values, transformation matrices, kernel filter values, and the like. In some embodiments, the memory 104 can be communicatively coupled to the image demosaicing circuit 106. In some embodiments, the memory 104 can be communicatively coupled to one or more circuits of the image demosaicing circuit 106.

[0014] The image demosaicing circuit 106 may contain one or more circuits and / or circuit groups for performing image demosaicing with minimal artifacts. For example, the image demosaicing circuit 106 may contain a raw interpolation circuit 108, a chroma estimator circuit 110, a luma estimator circuit 112, and a color space transformation circuit 114. In some embodiments, the image demosaicing circuit 106 may contain the luma zipper filter circuit 116 and / or the luma direction enhancer circuit 118. One or more circuits of the image demosaicing circuit 106 may be communicatively coupled, and the output of one of the circuits may be provided as input for another circuit. In some embodiments, a first circuit group may be communicatively coupled to a second circuit group. For example, the raw interpolation circuit 108 can be a color filter array (CFA) (e.g.from memory 104) and perform interpolation to provide a color value (e.g., green, G) at each position in the CFA. The fully available green values ​​from the raw interpolation circuit 108 can be provided as part of the input for the chroma estimation circuit 110. In some embodiments, the output of one circuit can be stored in memory 104, and another circuit can load the previous output from memory 104 for further processing.

[0015] The raw interpolation circuit 108 can perform interpolation on an input CFA to provide one or more color values ​​at each position of the CFA. For example, a CFA with an RGGB Bayer filter pattern can be received as input for the raw interpolation circuit 108. In some embodiments, a different Bayer filter pattern can be used. The CFA can have a color value (e.g., R, G, or B) at each pixel position. The raw interpolation circuit 108 can interpolate one of the color values ​​(e.g., G) (e.g., by linear interpolation) so that the selected color has a value at each pixel position of the CFA.

[0016] In some embodiments, directional interpolation is performed. For example, to determine the direction of a given pixel P, a patch of raw color values ​​from the CFA centered on P can be smoothed using a bilinear low-pass filter (LPF). The absolute value of the differences between second-order derivatives of the smoothed pixel values ​​can be determined in a horizontal and a vertical direction. The direction with the higher value can be considered the direction of pixel P. Based on the determined direction of pixel P, the interpolated color value at that position can be calculated by assigning more weight to the neighboring color values ​​along the same direction.

[0017] After interpolation, the raw interpolation circuit 108 can output a modified CFA where each pixel position has a value for a specific color (e.g., G). The modified CFA with fully available color values ​​(of at least one single color) can be provided to the chroma estimation circuit 110.

[0018] The chroma estimation circuit 110 can convert the fully available color values ​​from the first color space (e.g., the RGB color space) into an intermediate color space (e.g., a YiUiVi color space). Based on the fully available color values ​​calculated by the raw interpolation circuit 108, a first color chroma value in the intermediate color space can be calculated for each pixel position. Different formulas can be used for different pixel positions of the CFA, depending on which color values ​​are available at that position. For example, a first chroma value for pixels with an R raw color value can be calculated using a first formula, while a second chroma value for pixels with a B raw color value can be calculated using a second formula. In some embodiments, both formulas depend on the G raw color value or the calculated G color value at the same position.

[0019] For example, a Ui value for each pixel position of the CFA that has a B raw color value can be calculated using the following formula: Ui = (BG) / 2. A Vi value can be calculated for each pixel position of the CFA that has an R raw color value using the following formula: Vi = (RG) / 2. Interpolation and / or upscaling can be used to provide a Ui value and a Vi value at each pixel position of the CFA based on the neighboring calculated Ui and Vi values. The Ui and Vi values ​​can be smoothed to obtain Uc and Vc using a low-pass filter (LPF).

[0020] In some embodiments, the Ui and Vi values ​​can be interpolated and smoothed simultaneously using a polyphase filter. For example, a 4-phase polyphase filter can be determined based on an LPF kernel. Each phase of the filter can correspond to a different pattern of values ​​surrounding the value to be interpolated and / or smoothed. For example, if the pixel to be interpolated and / or smoothed is located at a position with a B raw color value, a first phase of the polyphase filter can be used to compute the smoothed Uc value at that position. (See Fig. 3A). If the pixel to be interpolated and / or smoothed has B raw color values ​​on the left and right, a second phase of the polyphase filter can be used to calculate the smoothed Uc value at that point. (See Fig. 3B). If the pixel to be interpolated and / or smoothed has B raw color values ​​above and below, a third phase of the multi-phase filter can be used to calculate the smoothed Uc value at that position. (See Fig. 3C). If the pixel to be interpolated and / or smoothed has B raw color values ​​at its corners, a fourth phase of the polyphase filter can be used to calculate the smoothed Uc value at that position. (See Fig. 3D). A similar procedure can be used to calculate the smoothed Vc values.

[0021] In some implementations, it may be useful to trim some of the values ​​used in the polyphase filter calculation. Removing outliers can preserve the colors of small features while attenuating chroma artifacts. For example, the values ​​used in the polyphase filter calculation can be sorted in a list (e.g., from lowest to highest or highest to lowest). The median value in the middle of the list can be determined. One or more values ​​in the list that meet an outlier criterion (e.g., the lowest value(s), the highest value(s), values ​​above (or below) a predefined threshold, etc.) can be modified to the median value. The polyphase filter calculation can then continue using the modified value(s).

[0022] If no values ​​are changed, the trimmed polyphase filter can be equivalent to an upsampling filter. If all values ​​are changed, the trimmed polyphase filter can be equivalent to a median filter.

[0023] The Chroma estimating circuit 110 can output a CFA with fully available Ui and Vi values ​​(or Uc and Vc values). The output can be stored in memory 104 or provided as input for the Luma estimating circuit 112.

[0024] Luma Estimator 112 can calculate a luma value (Yi) within the intermediate color space for each pixel position of the CFA based on the calculated chroma values ​​from Chroma Estimator 110 and the CFA raw color values ​​in the first color space. For example, the color sample value of a given pixel can be combined with the calculated Ui and Vi values ​​for that pixel to obtain the Yi value for the pixel. In some embodiments, different weights are applied to the Ui and Vi values ​​(e.g., chroma correction) based on the pixel's color in the CFA. For example, the Yi value for a pixel with an R raw color value can be calculated using the following formula: Yi = R + 0.5Ui - 1.5Vi. The Yi value for a pixel with a G raw color value can be calculated using the following formula: Yi = G + 0.5Ui + 0.5Vi.The Yi value for a pixel with a B raw color value can be calculated using the following formula: Yi = B - 1.5Ui + 0.5Vi. In some embodiments, the smoothed values ​​(e.g., Uc, Vc) are used instead of the intermediate values ​​(e.g., Ui, Vi) to obtain a luma estimate (e.g., Yc). The calculated luma values ​​Yi (or Yc) can be stored in memory 104 and / or fed to another circuit of the image demosaicing circuit 106 (e.g., the luma zipper filter circuit 116, the luma direction enhancement circuit 118, or the color space transformation circuit 114).

[0025] In some embodiments, the luma zipper filter circuit 116 can be used to apply a luma zipper filter to the luma intermediate values ​​(e.g., Yi values, Yc values) to remove (or reduce) zipper artifacts that would appear in the full-color image. The luma zipper filter can be a bilateral low-pass filter (LPF). In some embodiments, the luma zipper filter is a two-dimensional filter, so that the output is a linear combination of a window of input intermediate luma values ​​weighted by coefficients of a fixed filter kernel and coefficients of a dynamic range kernel, which depend on the input range and the input intermediate luma values. The fixed filter kernel can have a wide passband to preserve high-frequency content (e.g., textures, image details, etc.) in the full-color image.In some embodiments, the improved value of a single intermediate luma value can be calculated based on a 9x9 patch of input intermediate luma values.

[0026] For example, in some embodiments, the zipper-filtered luma value Yd can be calculated using the following equation: Yd=∑i,j=−44(Yc(i,j)F(i,j)R(i,j))∑i,j=−44(F(i,j)R(i,j)) where Yc(i,j) can be a patch of (smoothed) luma values ​​(e.g., a 9x9 patch), F(i,j) can be a spatial kernel, and R(i,j) can be a range kernel. In some embodiments, the range kernel can be a function that depends on the luma values ​​Yc and a set of central luma values ​​(e.g., values ​​of Yc(i,j) where -1 ≤ i ≤ 1 and -1 ≤ j ≤ 1). An input range can be computed based on the central luma values. If a particular value of Yc(i,j) lies within the input range, R(i,j) can be equal to 1. Conversely, if Yc(i,j) lies far outside the input range (e.g., due to exceeding a predefined threshold, etc.), R(i,j) can be equal to 0. The values ​​in between can decrease.

[0027] The calculated Yd-Luma values ​​can be stored in memory 104 and / or provided to another circuit of the image demosaicing circuit 106 (e.g. the Luma direction improvement circuit 118 or the color space transformation circuit 114).

[0028] In some embodiments, the luma direction enhancement circuit 118 can apply a direction filter to the intermediate luma values ​​(e.g., Yi values, Yc values, Yd values) to smooth edges in the final full-color image. The direction filter can start with a 9x9 patch of raw values ​​(e.g., from the CFA) and apply a bilinear LPF to compute a 7x7 patch of smoothed luma values. Edge directions can be calculated within a 5x5 patch of the smoothed luma values ​​by taking the absolute value of differences between second-order derivatives of the smoothed luma values ​​in the horizontal and vertical directions. (See Fig. 4) The horizontal edge value can be calculated, for example, using the following formula: Gh(i,j) = |Ys(i-1,j)-2*Ys(i,j)+Ys(i+1,j). The vertical edge value can be calculated using the following formula: Gv(ij) = |Ys(i,j-1)-2*Ys(i,j)+Ys(i,j+1). The local direction of a given pixel can be horizontal (with a value of 1) if Gh(i,j) < Gv(ij), or vertical (with a value of -1) otherwise.

[0029] The edge strength S(M) can be determined by applying a normalized LPF kernel to the calculated directions (e.g., L(i,j) values) and summing the values. A large positive value may indicate a strong horizontal edge, and a large negative value may indicate a strong vertical edge. The luma value for the pixel (e.g., Yd) can then be modified based on the determined direction. For example, a horizontal fit value can be calculated using the following formula: Yhor = (Yc(-1,0)+2*Yc(0,0)+Yc(1,0)) / 4. A vertical fit value can be calculated using the following formula: Yver = (Yc(0,-1)+2*Yc(0,0)+Yc(0,1)) / 4. The luma value Yd can be modified (e.g. by combining with, adding to, subtracting from, etc.) by (Yhor-Yd) * S(M) (or (Yver-Yd) * S(M) for vertical edges) to obtain Ye.

[0030] The calculated Ye-Luma values ​​can be stored in memory 104 and / or fed to another circuit of the image demosaicing circuit 106 (e.g. the color space transformation circuit 114).

[0031] The color space transformation circuit 114 can convert the resulting intermediate values ​​(e.g., YiUiVi, YcUcVc, YdUcVc, YeUcVc) into the output color space (e.g., YUV) using one or more transformation operations. For example, the intermediate values ​​can be multiplied by one or more transformation matrices to obtain the output values. The output of the color space transformation circuit 114 can be a full-color image of the original raw CFA and contain minimal demosaicing artifacts.

[0032] Fig. Figure 2 is a block diagram of an exemplary data flow 200 for demosaicing color samples to obtain a full-color output, according to at least one embodiment. In some embodiments, the data flow 200 represents the above in relation to Fig. 1 describes an image demosaicing method. A Bayer sample 202 can contain one or more pixels, each of which has at least one color value within a first color space (e.g., the RGB color space). The color values ​​of a single color (e.g., green values ​​204) can be interpolated using a directional interpolation 206, so that each pixel has a corresponding value of that color (e.g., fully available green values ​​208).

[0033] A chroma estimation 212 can combine the Bayer sample 202 and fully available green values ​​208 to obtain chroma values ​​210. For example, Ui can be calculated for pixels with unprocessed B color values ​​using the formula Ui = (BG) / 2 (or Ui = (B-Gi) / 2). Then, a chroma interpolation 218 can be performed to make Ui available at all pixel positions (e.g., to obtain fully available chroma values ​​222). In the chroma interpolation 218, a polyphase filter can be used to interpolate and / or upsample the Ui values ​​at all pixel positions. In some embodiments, a trimmed polyphase filter is used. The polyphase filter can smooth the Ui values ​​to obtain Uc values.

[0034] Similarly, chroma estimation 216 can combine Bayer samples 202 and fully available green values ​​208 to obtain chroma values ​​214. For example, Vi can be calculated at pixels with unprocessed R color values ​​using the formula Vi = (RG) / 2 (or Vi = (R-Gi) / 2). Then, chroma interpolation 220 can be performed to make Vi available at all pixel positions (e.g., to obtain fully available chroma values ​​224). Chroma interpolation 220 can use a polyphase filter to interpolate and / or upsample the Vi values ​​at all pixel positions. In some embodiments, a trimmed polyphase filter is used. The polyphase filter can smooth the Vi values ​​to obtain Vc values.

[0035] A luma estimate 226 can receive fully available chroma values ​​222 and fully available chroma values ​​224 to calculate luma values ​​228. Luma estimate 226 can also receive Bayer samples 202. The luma estimate 226 can calculate luma values ​​228 using the following formulas. For pixels with R color raw values: Yi = R + 0.5Ui - 1.5Vi. For pixels with G color raw values: Yi = G + 0.5Ui + 0.5Vi. For pixels with B color raw values: Yi = B - 1.5Ui + 0.5Vi. In some embodiments, the smoothed values ​​(e.g., Uc, Vc) are used instead of the intermediate values ​​(e.g., Ui, Vi) to obtain a luma estimate (e.g., Yc).

[0036] The color space transformation 230 can combine luma values ​​228, fully available chroma values ​​222, and fully available chroma values ​​224 to produce output values ​​232. Output values ​​232 can represent the output full-color image. The output values ​​232 can include values ​​for the luma component 234 of the output color space, values ​​for a first chroma component 236 of the output color space, and values ​​for a second chroma component 238 of the output color space. In some embodiments, the color space transformation 230 transforms luma values ​​228, fully available chroma values ​​222, and fully available chroma values ​​224 using one or more transformation matrices.

[0037] Fig. Figure 3A is a block diagram of an exemplary kernel (Phase-0 kernel 302) of a multiphase filter according to at least one embodiment. The Phase-0 kernel 302 can be based on a 5x5 LPF kernel. Although the Phase-0 kernel 302 in Fig. Since 3A is not normalized, a normalized LPF kernel can be used instead in some embodiments. The Phase 0 kernel 302 can be used during chroma estimation to interpolate and / or upscale and / or smooth Ui / Vi values ​​to adjacent pixel positions if the color value to be interpolated (e.g., B for Ui, R for Vi) is in the center of the kernel.

[0038] Fig. Figure 3B is a block diagram of an exemplary kernel (Phase 1 Kernel 304) of a multiphase filter according to at least one embodiment. The Phase 1 Kernel 304 can be based on a 5x5 LPF kernel. Although the Phase 1 Kernel 304 in Fig. Since 3B is not normalized, a normalized LPF kernel can be used instead in some embodiments. The Phase 1 kernel 304 can be used during chroma estimation to interpolate and / or upscale and / or smooth Ui / Vi values ​​to adjacent pixel positions when the color value to be interpolated (e.g., B for Ui, R for Vi) is located to the left and right of the pixel to be interpolated.

[0039] Fig. Figure 3C is a block diagram of an exemplary kernel (Phase-2 kernel 306) of a multiphase filter according to at least one embodiment. The Phase-2 kernel 306 can be based on a 5x5 LPF kernel. Although the Phase-2 kernel 306 in Fig. Since 3C is not normalized, a normalized LPF kernel can be used instead in some embodiments. The Phase 2 kernel 306 can be used during chroma estimation to interpolate and / or upscale and / or smooth Ui / Vi values ​​to adjacent pixel positions when the color value to be interpolated (e.g., B for Ui, R for Vi) is located at the top and bottom edges of the pixel to be interpolated.

[0040] Fig. Figure 3D is a block diagram of an exemplary kernel (Phase-4 kernel 308) of a multiphase filter according to at least one embodiment. The Phase-4 kernel 308 can be based on a 5x5 LPF kernel. Although the Phase-4 kernel 308 in Fig. If 3D is not normalized, a normalized LPF kernel can be used instead in some embodiments. The Phase 4 kernel 308 can be used during chroma estimation to interpolate and / or upscale and / or smooth Ui / Vi values ​​to adjacent pixel positions when the color value to be interpolated (e.g., B for Ui, R for Vi) is located at the corners of the pixel to be interpolated.

[0041] Fig. Figure 4 is a block diagram of an exemplary data flow 400 for luma direction enhancement according to at least one embodiment. As described above, luma values ​​(e.g., Vi values, Yc values, Yd values) can be enhanced directionally to reduce artifacts (e.g., for edge smoothing) in the full-color output image. The direction filter can start with a 9x9 patch of raw values ​​402 (e.g., from the CFA). A bilinear LPF 408 can be applied to unprocessed or raw values ​​402 to obtain a 7x7 patch of smoothed luma values ​​404. Subsequently, direction detection 410 can be performed to calculate edge directions within the smoothed luma values ​​404 and obtain directional luma values ​​406. Each local direction L can be based on the absolute value of the differences between the second-order derivatives of the smoothed luma values ​​in the horizontal and vertical directions.The horizontal edge value can be calculated, for example, using the following formula: Gh(i,j) = |Ys(i-1,j)-2*Ys(i,j)+Ys(i+1,j). The vertical edge value can be calculated using the following formula: Gv(ij) = |Ys(i,j-1)-2*Ys(i,j)+Ys(i,j+1). The local direction L of a given pixel can be horizontal (with a value of 1) if Gh(i,j) < Gv(ij) or vertical (with a value of -1) otherwise.

[0042] In some embodiments, direction detection can be performed during one or more of the methods described above (e.g., during chroma estimation, during luma estimation, during luma direction enhancement, etc.).

[0043] Fig. Figure 5 is a flowchart of an exemplary method 500 for demosaicing images with minimal artifacts, according to at least one embodiment. The method 500 can be performed using one or more processing units (e.g., central processing units (CPUs), graphics processing units (GPUs), accelerators, physics processing units (PPUs), data processing units (DPUs), etc.) that can contain (or communicate with) one or more storage devices. In at least one embodiment, the method 500 can be performed using one or more processing units. In at least one embodiment, the method 500 can be performed using one or more processing units of the computer system 600. In at least one embodiment, the method 500 can be performed by the image demosaicing system 102 of Fig. 1. In at least one embodiment, processing units performing the method 500 can execute instructions stored on a non-volatile, computer-readable storage medium. In at least one embodiment, the method 500 can be performed using multiple processing threads (e.g., CPU threads and / or GPU threads), with individual threads performing one or more individual functions, routines, subroutines, or operations of the method. In at least one embodiment, processing threads implementing the method 500 can be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, threads implementing the method 500 can be executed asynchronously. Various operations of the method 500 can be performed in a different order than that described in Fig. The operation is performed in the sequence shown in section 5. Some operations of method 500 can be performed simultaneously with other operations. In at least one embodiment, one or more of the operations shown in Fig. The 5 operations shown may not always be performed.

[0044] With reference to Fig. In Block 502, processing units executing Method 500 can calculate a color value within a first color space for a first pixel of a plurality of pixels, at least on the basis of color values ​​within a first group of pixels adjacent to the first pixel. In some embodiments, the pixels of the first group of adjacent pixels are part of the plurality of pixels. In some embodiments, processing units can determine a gradient direction of the first pixel and calculate the color value within the first color space for the first pixel of the plurality of pixels, at least on the basis of the color values ​​within the first color space of the first group of adjacent pixels along the gradient direction of the first pixel.

[0045] In block 504, processing units can calculate an initial chrominance value in a second color space for the first pixel, based at least on the calculated hue value in the first color space. In block 506, processing units can calculate a luminance value in the second color space for the first pixel, based at least on the initial chrominance value in the second color space.

[0046] In some embodiments, processing units in block 508 can apply a polyphase filter to the first chrominance value within the second color space to obtain a smoothed chrominance value. In some embodiments, at least one value of the polyphase filter is modified based on the fulfillment of an outlier criterion. For example, a specific phase of the polyphase filter can be selected, and N chrominance values ​​around the chrominance value to be smoothed can be considered. The N values ​​can be sorted by their value (e.g., by their magnitude), and one or more of the N values ​​that fulfill an outlier criterion (e.g., the highest value(s), the lowest value(s), etc.) can be modified. In some embodiments, one or more outlier values ​​can be set to the median value of the N values.

[0047] In some embodiments, processing units performing method 500 can modify the luminance value within the second color space for the first pixel in block 510 based on a luma zipper filter based on a combination of luminance values ​​within the second color space of a second group of pixels adjacent to the first pixel, a spatial filter kernel, and a range kernel.

[0048] In some embodiments, processing units in block 512 can determine a gradient direction and gradient strength for the first pixel. In block 514, processing units can modify the luminance value within the second color space for the first pixel based on the determined gradient direction and gradient strength.

[0049] In Block 516, processing units executing Procedure 500 can convert the luminance value within the second color space and the first chrominance value within the second color space into an output pixel value within a third color space. To convert the luminance value within the second color space and the first chrominance value within the second color space into an output pixel value within a third color space, processing units can apply a transformation matrix to the luminance value within the second color space and the first chrominance value within the second color space.

[0050] Fig. Figure 6 is a block diagram illustrating an exemplary computer system according to at least one embodiment of the present disclosure. The computer system 600 comprises the image demosaicing system 102 of the Fig.1. The Computer System 600 can function as a server or endpoint machine in an endpoint-server network environment, or as a peer device in a peer-to-peer (or distributed) network environment. The machine can be a television, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web device, a server, a network router, a switch or bridge, or any other machine capable of executing a set of instructions (sequential or otherwise) that specify the actions to be performed by that machine.Furthermore, although only a single machine is depicted, the term "machine" is also understood to encompass any collection of machines that, individually or collectively, execute a set (or sets) of instructions to perform one or more of the procedures discussed herein.

[0051] The exemplary computer system 600 comprises a processing unit (processor) 602, a main memory 604 (e.g., a read-only memory (ROM), a flash memory, a dynamic read / write memory (DRAM) such as a synchronous DRAM (SDRAM), a DDR-SDRAM, or a DRAM (RDRAM), etc.), a static memory 606 (e.g., flash memory, static RAM memory (SRAM), etc.), and a data storage unit 616, which communicate with each other via a bus 628.

[0052] The processor (processing unit) 602 represents one or more general-purpose processing units, such as a microprocessor, a central processing unit, or the like, and may include processing logic 622. In particular, the processor 602 may be a CISC (Complex Instruction Set Computing) microprocessor, a RISC (Reduced Instruction Set Computing) microprocessor, a VLIW (Very Long Instruction Word) microprocessor, or a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. The processor 602 may also be one or more special-purpose units, such as an application-specific integrated circuit (ASIC), a user-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. The processor 602 is configured to execute instructions 626 (e.g.,(to generate threat indicator alerts) in order to perform the operations described herein.

[0053] The computer system 600 may also include a network interface device 608. The computer system 600 may also include a display 610 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an input device 612 (e.g., a keyboard and an alphanumeric keypad, a motion-sensitive input device, a touchscreen), a cursor control device 614 (e.g., a mouse), and a signal generation device 618 (e.g., a loudspeaker). In some embodiments, the computer system 600 may not include a display 610, an input device 612, and / or a cursor control device 614 (e.g., in a headless configuration).

[0054] The data storage device 616 can contain a non-volatile, machine-readable storage medium 624 (also called a computer-readable storage medium) on which one or more sets of instructions 626 (e.g., for demosaicing color samples into full-color images) are stored, embodying one or more of the procedures or functions described herein. During their execution by the computer system 600, the instructions 626 can also reside wholly or at least partially in the main memory 604 and / or the processor 602, the main memory 604 and the processor 602 also being machine-readable storage media. Furthermore, the instructions can be transmitted or received via the network interface 608 over a network 620.

[0055] In one implementation, instructions 626 include instructions for demosaicing color samples into full-color images. While the computer-readable storage medium 624 (machine-readable storage medium) is represented as a single medium in an exemplary implementation, the terms "computer-readable storage medium" and "machine-readable storage medium" should be understood to include a single medium or multiple media (for example, a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions. The terms "computer-readable storage medium" and "machine-readable storage medium" are also intended to include any media capable of storing, encoding, or transmitting a set of instructions for execution by the machine and causing the machine to execute one or more of the procedures described in this disclosure.The terms "computer-readable storage medium" and "machine-readable storage medium" therefore include, but are not limited to, solid-state storage, optical media and magnetic media.

[0056] Other variations are within the spirit of the present disclosure. Thus, while various modifications and alternative constructions can be made with respect to the disclosed methods, certain illustrated embodiments are shown in the drawings and have been described in detail above. However, it is understood that the intention is not to limit the disclosure to the specific disclosed form or forms, but rather, on the contrary, to cover all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the disclosure as defined in the attached claims.

[0057] The use of the terms "a," "an," "the," and similar referents in the context of describing disclosed embodiments (particularly in the context of the following claims) is to be interpreted as covering both the singular and the plural unless otherwise specified herein or the context clearly contradicts this, and not as defining an expression. The terms "comprising," "having," "including," and "containing" are to be interpreted as open expressions (i.e., in the sense of "including without being limited to") unless otherwise specified. "Connected" is to be interpreted as partially or completely contained within, attached to, or joined to one another when used unmodified and referring to physical connections, even if an element is inserted between them.The mention of value ranges herein is intended merely as a quick method of individually referring to each separate value falling within the range, unless otherwise stated herein, and each separate value is included in the description as if it were individually reproduced herein. The use of the term "set" (e.g., "a set of objects") or "subset" in at least one embodiment is to be understood as a non-empty compilation comprising one or more elements, unless otherwise noted or the context contradicts this. Furthermore, unless otherwise stated or the context contradicts this, the term "subset" of a corresponding set does not necessarily mean a proper subset of the corresponding set, but the subset and the corresponding set may be the same.

[0058] Unless specifically stated otherwise or the context clearly contradicts it, connective language, such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C," is otherwise to be understood in the context in which it is generally used to indicate that an object, expression, etc., can be either A, B, or C, or any non-empty subset of the sentence consisting of A, B, and C. For example, in the illustrated example of a sentence containing three elements, the connective phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any one of the following: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}.Thus, such connecting expressions should generally not express that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by the context, the term "multiplicity" also denotes a state of plurality (e.g., "a multitude of elements" denotes multiple elements). In at least one embodiment, a multitude consists of at least two elements, but may also include more if this is either explicitly stated or indicated by the context. Furthermore, unless otherwise stated or evident from the context, the phrase "based on" means "at least partially based on" or "at least based on," and not "exclusively based on."

[0059] The operations of processes described herein may be performed in any suitable order, unless otherwise specified herein or the context clearly precludes it. In at least one embodiment, a process, such as the processes described herein (or variations and / or combinations thereof), is carried out under the control of one or more computer systems configured with executable instructions, and is implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, by hardware or combinations thereof. In at least one embodiment, code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that can be executed by one or more processors.In at least one embodiment, a computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transitory signals (e.g., a propagating transient electrical or electromagnetic transmission) but includes non-transitory data storage circuits (e.g., buffers, caches, and queues) within transient signal senders / receivers. In at least one embodiment, the code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media containing executable instructions (or other storage for executable instructions) which, when executed (i.e., as a result of execution) by one or more processors of a computer system, cause the computer system to perform the operations described herein.In at least one embodiment, a set of nontransitory computer-readable storage media comprises multiple nontransitory computer-readable storage media. One or more of the individual nontransitory storage media do not contain the entire code, while multiple nontransitory computer-readable storage media collectively store the entire code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors—for example, a nontransitory computer-readable storage medium stores instructions, and a central processing unit (CPU) executes some of the instructions, while a graphics processing unit (GPU) executes other instructions.In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

[0060] Accordingly, computer systems in at least one embodiment are configured to implement one or more services that, individually or collectively, perform operations of the processes described herein, and such computer systems are configured with applicable hardware and / or software that enables the execution of operations. Furthermore, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment, a distributed computer system comprising several devices that operate differently, such that the distributed computer system performs the operations described herein and such that a single device does not perform all operations.

[0061] The use of any and all examples or illustrative language (e.g., "such as") provided in this document is intended solely to better clarify the embodiments of the disclosure and does not constitute a limitation of the scope of the disclosure unless claimed otherwise. No wording in the description should be interpreted as indicating any unclaimed element as essential to the implementation of the disclosure.

[0062] Any references, including publications, patent applications and patents mentioned in this document, are hereby incorporated by reference to the same extent as if each reference had been individually and specifically indicated as being included by reference and set forth in this document in its entirety.

[0063] In the description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It is understood that these terms are not intended to be synonymous. Rather, in specific examples, "connected" or "coupled" can be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" can also mean that two or more elements are not in direct contact with each other, but nevertheless interact or work together.

[0064] Unless expressly stated otherwise, in some embodiments, terms such as "processing", "calculating", "calculating", "determining" or the like throughout the description are understood to refer to actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or convert data represented as physical, e.g. electronic, quantities in the registers and / or memories of the computing system into other data represented in a similar manner as physical quantities in the memories, registers or other such information storage, transmission or display devices of the computing system.

[0065] Similarly, the term "processor" can refer to any device or section of a device that processes electronic data from registers and / or memories and converts that electronic data into other electronic data that can be stored in registers and / or memories. As non-restrictive examples, the "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, "software" processes can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Furthermore, each process can refer to multiple processes for executing instructions sequentially or in parallel, continuously or intermittently.The terms “system” and “method” are used interchangeably in this document for at least one embodiment insofar as a system can embody one or more methods and the methods can be considered as a system.

[0066] This document may refer to the acquisition, capture, reception, or input of analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, a method for acquiring, capturing, receiving, or inputting analog and digital data can be accomplished in various ways, such as receiving data as a parameter of a function call or a call to an application programming interface. In some implementations, the process of acquiring, capturing, receiving, or inputting analog or digital data can be achieved by transmitting data over a serial or parallel interface.In another implementation, processes of acquiring, capturing, receiving, or inputting analog or digital data can be accomplished by transmitting data over a computer network from the providing entity to the capturing entity. At least one embodiment can also refer to providing, outputting, transmitting, sending, or displaying analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or displaying analog or digital data can be accomplished by transmitting data as an input or output parameter of a function call, a parameter of an application programming interface, or an interprocess communication mechanism.

[0067] Although the descriptions presented here are exemplary embodiments of the described methods, other architectures may also be used to implement the described functionality, and these are intended to be within the scope of this disclosure. Furthermore, although specific distributions of responsibilities are defined above for the purpose of description, various functions and responsibilities could be distributed and divided differently depending on the circumstances.

[0068] Although the subject matter has been further described in language specific to structural features and / or process steps, it is understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or steps described. Rather, specific features and steps are disclosed as exemplary ways of implementing the claims.

Claims

[1] Circuit comprising: a first circuit: to compute a color value within a first color space for a first pixel of a plurality of pixels based on at least color values ​​within the first color space of a first group of pixels adjacent to the first pixel, wherein the pixels of the first group of adjacent pixels are part of the plurality of pixels; and to provide the color value within the first color space for a second circuit; wherein the second circuit is communicatively coupled to the first circuit, wherein the second circuit is designed as follows: to calculate a first chrominance value in a second color space for the first pixel, at least on the basis of the calculated color value in the first color space; and to provide the first chrominance value in the second color space of a third circuit; wherein the third circuit is communicatively coupled with the second circuit, wherein the third circuit is designed as follows: to calculate a luminance value within the second color space for the first pixel, at least based on the first chrominance value within the second color space; and to provide the luminance value within the second color space of a fourth circuit; and wherein the fourth circuit is communicatively coupled with the third circuit, wherein the fourth circuit is designed as follows: to convert the luminance value within the second color space and the first chrominance value within the second color space into an output pixel value within a third color space. [2] Circuit according to claim 1, wherein the first circuit for calculating the color value within the first color space for the first pixel of the plurality of pixels is designed on the basis of at least the color values ​​within the first color space of the first group of pixels adjacent to the first pixel,: to determine a gradient direction of the first pixel; and to calculate the color value within the first color space for the first pixel of the plurality of pixels, at least on the basis of the color values ​​within the first color space of the first group of neighboring pixels along the gradient direction of the first pixel. [3] Circuit according to claim 1 or 2, wherein the third circuit is further configured to apply a multiphase filter to the first chrominance value within the second color space in order to obtain a smoothed chrominance value. [4] Circuit according to claim 3, wherein at least one value of the multiphase filter is modified on the basis of fulfilling an outlier criterion. [5] Circuit according to one of the preceding claims, wherein the third circuit is further configured to modify the luminance value within the second color space for the first pixel on the basis of a luma zipper filter based on a combination of luminance values ​​within the second color space of a second group of pixels adjacent to the first pixel, a spatial filter kernel and a range kernel. [6] Circuit according to one of the preceding claims, wherein the third circuit is further configured as follows: to determine a gradient direction and gradient strength of the first pixel; and to modify the luminance value within the second color space for the first pixel based on the specified gradient direction and gradient strength. [7] Circuit according to one of the preceding claims, wherein the fourth circuit is configured to convert the luminance value within the second color space and the first chrominance value within the second color space into an output pixel value within the third color space in order to apply a transformation matrix to the luminance value within the second color space and the first chrominance value within the second color space. [8] System encompassing: a storage device; and one or more circuit groups that are communicatively coupled to the storage device, wherein the one or more circuit groups are designed as follows: to calculate a color value within a first color space for a first pixel of a plurality of pixels at least on the basis of color values ​​within the first color space of a first group of pixels adjacent to the first pixel, wherein the pixels of the first group of adjacent pixels are part of the plurality of pixels; to calculate a first chrominance value in a second color space for the first pixel, at least on the basis of the calculated color value in the first color space; to calculate a luminance value in the second color space for the first pixel, at least on the basis of the first chrominance value in the second color space; and to convert the luminance value in the second color space and the first chrominance value in the second color space into an output pixel value in a third color space. [9] System according to claim 8, wherein one or more circuit groups are configured to calculate the color value within the first color space for the first pixel of the plurality of pixels on the basis of at least the color values ​​within the first color space of the group of pixels adjacent to the first pixel: to determine a gradient direction of the first pixel; and to calculate the color value within the first color space for the first pixel of the plurality of pixels, at least on the basis of the color values ​​within the first color space of the first group of neighboring pixels along the gradient direction of the first pixel. [10] System according to claim 8 or 9, wherein one or more circuit groups are further configured to apply a multiphase filter to the first chrominance value within the second color space in order to obtain a smoothed chrominance value. [11] System according to claim 10, wherein at least one value of the polyphase filter is modified based on the fulfillment of an outlier criterion. [12] System according to one of claims 8 to 11, wherein one or more circuit groups are further configured to modify the luminance value within the second color space for the first pixel on the basis of a luma zipper filter based on a combination of luminance values ​​within the second color space of a second group of pixels adjacent to the first pixel, a spatial filter kernel and a range kernel. [13] System according to one of claims 8 to 12, wherein one or more circuit groups are further configured as follows: to determine a gradient direction and gradient strength of the first pixel; and to modify the luminance value within the second color space for the first pixel based on the specified gradient direction and gradient strength. [14] System according to one of claims 8 to 13, wherein, for converting the luminance value within the second color space and the first chrominance value within the second color space into an output pixel value within the third color space, one or more circuit groups are configured to apply a transformation matrix to the luminance value within the second color space and the first chrominance value within the second color space. [15] System encompassing: a storage device; and one or more circuit groups that are communicatively coupled to the storage device, comprising one or more circuit groups: a first circuit group: to compute a color value within a first color space for a first pixel of a plurality of pixels, at least on the basis of color values ​​within the first color space of a first group of pixels adjacent to the first pixel, wherein the pixels of the first group of adjacent pixels are part of the plurality of pixels; and to provide the color value within the first color space of a second circuit group; and wherein the second circuit group is communicatively coupled with the first circuit group, wherein the second circuit group is designed as follows: to calculate a first chrominance value in a second color space for the first pixel, at least on the basis of the calculated color value in the first color space of the first circuit group; to calculate a luminance value in the second color space for the first pixel, at least on the basis of the first chrominance value in the second color space; and to convert the luminance value within the second color space and the first chrominance value within the second color space into an output pixel value within a third color space. [16] System according to claim 15, wherein the first circuit group for calculating the color value within the first color space for the first pixel of the plurality of pixels is designed on the basis of at least the color values ​​within the first color space of the first group of pixels adjacent to the first pixel,: to determine a gradient direction of the first pixel; and to calculate the color value within the first color space for the first pixel of the plurality of pixels, at least on the basis of the color values ​​within the first color space of the first group of neighboring pixels along the gradient direction of the first pixel. [17] System according to claim 15 or 16, wherein the second circuit group is further configured to apply a multiphase filter to the first chrominance value within the second color space in order to obtain a smoothed chrominance value. [18] System according to claim 17, wherein at least one value of the multiphase filter is modified based on the fulfillment of an outlier criterion. [19] System according to one of claims 15 to 18, wherein the second circuit group is further configured to modify the luminance value within the second color space for the first pixel on the basis of a luma zipper filter based on a combination of luminance values ​​within the second color space of a second group of pixels adjacent to the first pixel, a spatial filter kernel and a range kernel. [20] System according to one of claims 15 to 19, wherein the second circuit group is further configured as follows: to determine a gradient direction and gradient strength of the first pixel; and to modify the luminance value within the second color space for the first pixel based on the specified gradient direction and gradient strength.

Images