Method and device for demosaicing image

Abstract

The embodiment of the invention provides a method and device for carrying out demosaicing processing on an image and a computer readable storage medium. The method may include: determining gradient data based on an input image, the input image including a plurality of pixels and having a Bayer format, the gradient data including a horizontal gradient value of each of the plurality of pixels along a horizontal direction of the input image and a vertical gradient value along a vertical direction of the input image; determining, based on the gradient data, direction data for representing whether each of the plurality of pixels is at a horizontal edge or a vertical edge of the input image; and based on the direction data, performing interpolation operation on the input image to obtain a group of output images corresponding to the input image, the group of output images including a green channel image, a blue channel image and a red channel image of the plurality of pixels.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular, to a method, an apparatus, and a computer-readable storage medium for performing de-mosaic processing on an image. Background Art

[0002] Due to various factors such as cost and volume, cameras in current electronic devices (such as mobile phones, security equipment, and vehicle-mounted equipment) usually use complementary metal oxide semiconductor (CMOS) sensors as image acquisition sensors. This sensor is generally covered with a color filter array arranged in a Bayer manner, so that only one of the three primary colors (such as red, green, and blue) is sampled for each pixel. In order to obtain a color image from this sampled image, demosaicing is usually required. Therefore, how to efficiently perform demosaicing has become one of the problems that need to be solved. Summary of the invention

[0003] In view of the need for improvement of the prior art, embodiments of the present disclosure provide a method, an apparatus, and a computer-readable storage medium for demosaicing an image.

[0004] On the one hand, an embodiment of the present disclosure provides a method for de-mosaicing an image, comprising: determining gradient data based on an input image, wherein the input image includes multiple pixels and has a Bayer format, and the gradient data includes a horizontal gradient value of each pixel of the multiple pixels along the horizontal direction of the input image and a vertical gradient value along the vertical direction of the input image; determining direction data based on the gradient data, wherein the direction data is used to indicate whether each pixel of the multiple pixels is at a horizontal edge or a vertical edge of the input image; and performing an interpolation operation on the input image based on the direction data to obtain a set of output images corresponding to the input image, wherein the set of output images includes a green channel image, a blue channel image, and a red channel image of the multiple pixels.

[0005] On the other hand, an embodiment of the present disclosure provides an apparatus for performing de-mosaic processing on an image, comprising: a gradient determination unit, configured to determine gradient data based on an input image, wherein the input image comprises a plurality of pixels and has a Bayer format, and the gradient data comprises a horizontal gradient value of each pixel of the plurality of pixels along a horizontal direction of the input image and a vertical gradient value along a vertical direction of the input image; a direction determination unit, configured to determine direction data based on the gradient data, wherein the direction data is used to indicate whether each pixel of the plurality of pixels is at a horizontal edge or a vertical edge of the input image; and an interpolation unit, configured to perform an interpolation operation on the input image based on the direction data to obtain a set of output images corresponding to the input image, wherein the set of output images comprises a green channel image, a blue channel image, and a red channel image of the plurality of pixels.

[0006] On the other hand, an embodiment of the present disclosure provides a device for de-mosaicing an image, comprising: at least one processor; a memory communicating with the at least one processor, on which executable code is stored, and when the executable code is executed by the at least one processor, the at least one processor executes the above method.

[0007] On the other hand, an embodiment of the present disclosure provides a computer-readable storage medium storing executable codes, which perform the above method when executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other objects, features and advantages of embodiments of the present disclosure will become more apparent through a more detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like elements throughout the various drawings.

[0009] Figure 1 is a schematic flow chart of a method for demosaicing an image according to some embodiments.

[0010] Figure 2 is a schematic flow chart of a process for demosaicing an image according to some embodiments.

[0011] Figure 3A An example of a portion of pixels of an input image is shown.

[0012] Figure 3B An example of a convolution kernel is shown.

[0013] Figure 3C An example of direction determination is shown.

[0014] Figure 3D Another example of a portion of pixels of an input image is shown.

[0015] Figure 3E An example of nine pixels is shown.

[0016] Figure 4 is a schematic block diagram of an apparatus for demosaicing an image according to some embodiments.

[0017] Figure 5 is a schematic block diagram of an apparatus for demosaicing an image according to some embodiments. DETAILED DESCRIPTION

[0018] The subject matter described herein will now be discussed with reference to various embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope of protection, applicability or examples set forth in the claims.

[0019] In many fields (such as mobile phones, security, and automotive fields), mainstream cameras currently used generally use CMOS sensors. CMOS sensors generally use Bayer filters to capture photons so that each pixel samples only one of the three primary colors of red, green, and blue. The image obtained in this way is generally considered to have a Bayer format. For example, in such a Bayer format image, typically 25% of the pixels have a red component, 25% of the pixels have a blue component, and 50% of the pixels have a green component. In order to obtain a full-color image from such an image with a Bayer format, it is usually necessary to perform a demosaicing process on the Bayer format image. The demosaicing process mainly estimates the missing color component value of each pixel in the Bayer format image, such as the red component value, the green component value, or the blue component value. So, how to efficiently and reliably implement the demosaicing process becomes one of the problems that need to be solved.

[0020] The embodiments of the present disclosure provide a technical solution for performing de-mosaic processing on an image, which will be described below in conjunction with specific embodiments.

[0021] Figure 1 is a schematic flow chart of a method for demosaicing an image according to some embodiments.

[0022] like Figure 1 As shown, in step 102, gradient data may be determined based on an input image.

[0023] The input image may include a plurality of pixels and have a Bayer format. For example, each pixel in the input image has one of a red component value, a green component value, or a blue component value. For example, in some implementations, 25% of the pixels in the input image have a red component value, 25% of the pixels have a blue component value, and 50% of the pixels have a green component value.

[0024] The gradient data may include a horizontal gradient value of each pixel of the plurality of pixels along a horizontal direction of the input image and a vertical gradient value along a vertical direction of the input image.

[0025] In step 104, direction data may be determined based on the gradient data.

[0026] The direction data may be used to indicate whether each of the plurality of pixels is at a horizontal edge or a vertical edge of the input image. It should be understood that the horizontal edge mentioned herein may indicate that the pixel is a pixel on the horizontal edge of the input image or a pixel near the horizontal edge; similarly, the vertical edge mentioned herein may indicate that the pixel is a pixel on the vertical edge of the input image or a pixel near the vertical edge.

[0027] In step 106, an interpolation operation may be performed on the input image based on the direction data to obtain a set of output images corresponding to the input image. The set of output images may include a green channel image, a blue channel image, and a red channel image of a plurality of pixels.

[0028] The input image may correspond to the output image, such as having the same resolution. For example, if the resolution of the input image is m*n, then the green channel image, the blue channel image, and the red channel image will also have a resolution of m*n.

[0029] In an embodiment of the present disclosure, gradient data for a plurality of pixels of an input image may be first determined, and direction data may be determined based on the gradient data. Based on the direction data, it may be determined whether each pixel is at a horizontal edge or a vertical edge of the input image. Based on such information, an interpolation operation may be performed on the input image to obtain each channel image. It can be seen that the entire process has low algorithmic complexity and low hardware cost requirements, and is therefore simple and efficient in implementation. Moreover, since the pixel is considered to be at a horizontal edge or a vertical edge when performing the interpolation operation, a good de-mosaicing effect may be achieved.

[0030] Gradient data can generally reflect the changes of each pixel along the horizontal and vertical directions of the input image. In the embodiments of the present disclosure, various applicable methods can be used to determine the gradient data. For example, in some embodiments, for each of the multiple pixels, the vertical gradient value of the pixel can be determined based on the pixel value of the pixel in the input image and the pixel values ​​of a group of pixels in the same column as the pixel in the input image; in addition, the horizontal gradient value of the pixel can be determined based on the pixel value of the pixel in the input image and the pixel values ​​of a group of pixels in the same row as the pixel in the input image.

[0031] As mentioned above, the input image has a Bayer format, and each pixel has one of a red component value, a green component value, or a blue component value. Therefore, it can be understood that the pixel value of a pixel in the input image can be one of a red component value, a green component value, or a blue component value.

[0032] In some cases, when calculating the vertical gradient value of each pixel, a group of pixels in the same column as the pixel may include several pixels (such as three pixels) above the pixel and several pixels (such as three pixels) below the pixel. For example, for a pixel with a red component value in the input image, when calculating the vertical gradient value of the pixel, two pixels with a green component value and one pixel with a red component value above the pixel and two pixels with a green component value and one pixel with a red component value below the pixel may be considered.

[0033] Similarly, when calculating the horizontal gradient value of each pixel, a group of pixels located in the same row as the pixel may include several pixels (such as three pixels) on the left side of the pixel and several pixels (such as three pixels) on the right side of the pixel. For example, for a pixel with a red component value in the input image, when calculating the horizontal gradient value of the pixel, two pixels with a green component value and one pixel with a red component value on the left side of the pixel and two pixels with a green component value and one pixel with a red component value on the right side of the pixel may be considered.

[0034] In some implementations, the gradient data may be represented as a gradient map. For example, the gradient data may include a horizontal gradient map and a vertical gradient map. The horizontal gradient map may include a horizontal gradient value of each pixel in a plurality of pixels along a horizontal direction of the input image. The vertical gradient map may include a vertical gradient value of each pixel in a plurality of pixels along a vertical direction of the input image.

[0035] After determining the gradient data, the direction data may be determined. In some embodiments, the direction data may be represented as a direction map. For ease of description, in this document, the direction map corresponding to the direction data is referred to as an available direction map, that is, a direction map that can be used for subsequent interpolation operations. The available direction map may represent the direction of each pixel in a plurality of pixels. The direction of a pixel may be a horizontal direction, a vertical direction, or no direction. If a pixel has a horizontal direction, it may represent that the pixel is at a horizontal edge of an input image; if a pixel has a vertical direction, it may represent that the pixel is at a vertical edge of an input image; if a pixel has no direction, it may represent that the pixel is neither at a horizontal edge nor at a vertical edge.

[0036] The available directional map can be determined in various applicable ways. For example, in some embodiments, a first directional map for a plurality of pixels can be generated based on the gradient data. Specifically, the direction of each pixel can be determined based on the horizontal gradient value and the vertical gradient value of each pixel, thereby generating the first directional map. Therefore, the first directional map can represent the direction of each pixel determined based on the gradient data.

[0037] For example, in some embodiments, for each pixel in a plurality of pixels, if the absolute difference between the vertical gradient value of the pixel and the horizontal gradient value of the pixel is greater than a preset gradient threshold, it is indicated that the pixel is at a certain edge. Specifically, if the difference between the vertical gradient value of the pixel minus the horizontal gradient value of the pixel is greater than the preset gradient threshold, it means that the pixel has a more obvious change in the vertical direction, and it can be determined that the pixel has a horizontal direction; if the difference between the horizontal gradient value of the pixel minus the vertical gradient value of the pixel is greater than the preset gradient threshold, it means that the pixel has a more obvious change in the horizontal direction, and it can be determined that the pixel has a vertical direction; if the absolute difference between the vertical gradient value of the pixel and the horizontal gradient value of the pixel is less than or equal to the preset gradient threshold, it can be determined that the pixel has no direction. The absolute difference here may refer to the absolute value of the difference between the vertical gradient value and the horizontal gradient value. The preset gradient threshold can be set according to factors such as actual application scenarios and business needs, and this document does not limit this.

[0038] In some cases, there may be some errors in the directionless judgment of pixels based on gradient data. In order to reduce such errors, the direction of the directionless pixels can be re-judged based on the adjacent pixel information of the pixels determined to be directionless, thereby further improving the accuracy and reliability of subsequent interpolation operations.

[0039] For example, based on the first direction map, a first set of non-directional pixels can be determined. Specifically, the first set of non-directional pixels can be the pixels among multiple pixels that are non-directional in the first direction map. The direction of the first set of non-directional pixels can be re-determined based on the adjacent pixel information of the first set of non-directional pixels, so as to generate an available direction map.

[0040] In some embodiments, the direction of the non-directional pixels can be corrected twice. For example, based on the horizontal gradient value and vertical gradient value of the adjacent pixels of the first set of non-directional pixels, the horizontal gradient value and vertical gradient value of the first set of non-directional pixels can be filtered respectively to determine the horizontal filtered gradient value and vertical filtered gradient value of the first set of non-directional pixels.

[0041] Various applicable filtering methods can be adopted. For example, taking each pixel in the first set of non-directional pixels as the center, convolution filtering can be performed on its horizontal gradient value and vertical gradient value respectively. Specifically, each pixel in the first set of non-directional pixels can correspond to the center of the convolution kernel. In this case, the adjacent pixels of each pixel in the first set of non-directional pixels can include eight pixels around the pixel, such as the pixels located above, below, to the left, to the right, in the upper left, in the upper right, in the lower left, and in the lower right of the pixel. The size of the convolution kernel and the size of each element can be set according to the actual application scenario, requirements, etc. For example, the size of the convolution kernel can be 3x3. This is not limited herein.

[0042] Based on the horizontal filtered gradient value and vertical filtered gradient value of the first set of non-directional pixels, the direction of the first set of non-directional pixels can be re-determined, and thus a second direction map can be obtained from the first direction map. For example, in the first direction map, the direction of the first non-directional pixel can be replaced with the re-determined direction of the first set of non-directional pixels, so as to obtain the second direction map.

[0043] The process of determining the pixel direction using the horizontal filtered gradient value and vertical filtered gradient value can be similar to the process of determining the pixel direction using the horizontal gradient value and vertical gradient value. For example, in some embodiments, for each pixel in the first set of non-directional pixels: if the difference between the vertical filtered gradient value of the pixel and the horizontal filtered gradient value of the pixel is greater than a preset filtered gradient threshold, it is determined that the pixel has a horizontal direction; if the difference between the horizontal filtered gradient value of the pixel and the vertical filtered gradient value of the pixel is greater than the preset filtered gradient threshold, it is determined that the pixel has a vertical direction; if the absolute difference between the vertical filtered gradient value and the horizontal filtered gradient value of the pixel is less than or equal to the preset filtered gradient threshold, it is determined that the pixel has no direction. Here, the absolute difference can represent the absolute value of the difference between the vertical filtered gradient value and the horizontal filtered gradient value. The preset filtered gradient threshold can be set according to factors such as actual business requirements and application scenarios.

[0044] It can be seen that since the horizontal filtering gradient value and the vertical filtering gradient value take into account the conditions of the adjacent or surrounding pixels of the non-directional pixel, based on such data, the direction of the pixel previously determined to be non-directional based on the gradient data can be corrected more accurately.

[0045] In order to further reduce the direction judgment error about the non-direction pixel, the direction of the non-direction pixel can be corrected again.

[0046] For example, a second group of non-directional pixels can be determined based on the second directional map. Specifically, the second group of non-directional pixels can be pixels with no direction in the second directional map among a plurality of pixels. The direction of the second group of non-directional pixels can be re-determined based on the adjacent pixel information of the second group of non-directional pixels, so as to obtain a usable directional map from the second directional map. For example, the direction of the second group of non-directional pixels can be replaced with the re-determined direction of the second group of non-directional pixels in the second directional map, so as to obtain a usable directional map. It can be seen that after such a process, the direction judgment of each pixel is made more accurate and reliable, thereby improving the accuracy of subsequent interpolation operations.

[0047] Various applicable methods may be used to re-determine the directions of the second group of non-directional pixels.

[0048] For example, for each pixel in the second group of non-directional pixels, if a group of neighboring pixels of the pixel have the same direction in the second directional map, the direction of the group of neighboring pixels in the second directional map can be determined as the direction of the pixel in the available directional map.

[0049] For example, the set of neighboring pixels may include pixels above, below, to the left, and to the right of the pixel, in which case the direction of the pixel in the available direction map may be the same as the directions of the neighboring pixels.

[0050] For another example, the group of adjacent pixels may include pixels above, to the upper left, and to the upper right of the pixel and pixels below, to the lower left, and to the lower right of the pixel. If these adjacent pixels have a horizontal direction in the second directional map, the direction of the pixel in the available directional map may be a horizontal direction.

[0051] For another example, the group of adjacent pixels may include pixels on the left, upper left, and lower left of the pixel and pixels on the right, upper right, and lower right of the pixel. If the group of adjacent pixels has a vertical direction in the second directional map, the direction of the pixel in the available directional map may be a vertical direction.

[0052] From the above content, it can be seen that the second directional map is obtained after adjusting the directions of the non-directional pixels in the first directional map, and the usable directional map is obtained after adjusting the directions of the non-directional pixels in the second directional map. It can be seen that through such a process, the directions of the non-directional pixels can be effectively corrected, and the judgment errors of the non-directional pixels can be reduced, thereby improving the accuracy and reliability of subsequent interpolation operations.

[0053] After determining the directional data, an interpolation operation may be performed based on the directional data to generate a set of output images.

[0054] For example, an interpolation operation may be performed on the input image based on the direction data to generate an initial green channel image, an initial blue channel image, and an initial red channel image of a plurality of pixels. The initial green channel image may include initial green component values ​​of a plurality of pixels, the initial blue channel image may include initial blue component values ​​of a plurality of pixels, and the red channel image may include initial red component values ​​of a plurality of pixels.

[0055] Outlier correction can be performed based on the initial green channel image, the initial blue channel image, and the initial red channel image to generate the green channel image, the blue channel image, and the red channel image mentioned in step 106. The green channel image may include green component values ​​of multiple pixels, the blue channel image may include blue component values ​​of multiple pixels, and the red channel image may include red component values ​​of multiple pixels. In this way, the images of the three channels finally obtained are more accurate.

[0056] As mentioned above, in the input image, each of the multiple pixels has only one of the color component values. For ease of description, the pixel values ​​of the multiple pixels in the input image are referred to as initial pixel values, and accordingly, the initial pixel value may be one of the initial red component value, the initial blue component value, or the initial green component value.

[0057] In some embodiments, an interpolation operation may be performed on a green channel of a first group of pixels in the plurality of pixels based on the direction data to generate an initial green channel image. The first group of pixels does not have an initial green component value in the input image.

[0058] Then, based on the initial green channel image, interpolation operations may be performed on the blue channel of the second group of pixels in the plurality of pixels and on the red channel of the third group of pixels in the plurality of pixels, respectively, so as to generate an initial blue channel image and an initial red channel image. The second group of pixels may be pixels that do not have initial blue component values ​​in the input image, and the third group of pixels may be pixels that do not have red component values ​​in the input image. The interpolation operation for the blue channel may be performed before or after the interpolation operation for the red channel, or the two operations may be performed in parallel, which is not limited herein.

[0059] In the embodiment of the present disclosure, the interpolation operation is first performed on the green channel, because there are usually more green component values ​​in the input image in the bayer format, for example, as mentioned above, 25% of the pixels of the input image may have red component values, 25% of the pixels may have blue component values, and 50% of the pixels may have green component values. In this way, the interpolation operation on the green channel is relatively easy.

[0060] In some embodiments, the interpolation operation performed on the green channel may be performed based on the direction data. For example, for each pixel in the first group of pixels: if the pixel is at the vertical edge of the input image, the initial green component value of the pixel may be determined based on the pixel value of the pixel and the pixel values ​​of the first group of neighboring pixels in the same column as the pixel; if the pixel is at the horizontal edge of the input image, the initial green component value of the pixel may be determined based on the pixel value of the pixel and the pixel values ​​of the second group of neighboring pixels in the same row as the pixel; if the pixel is not at the vertical edge and the horizontal edge of the input image, the initial green component value of the pixel may be determined based on the pixel values ​​of the third group of neighboring pixels around the pixel.

[0061] For example, the first group of neighboring pixels may include at least two pixels with initial green component values ​​that are in the same column and adjacent to the pixel in the input image and at least two pixels with the same color component value as the pixel, such as two pixels with initial green component values ​​above and below the pixel, respectively, and two pixels with the same color component value as the pixel, respectively above and below the pixel.

[0062] Similarly, the second group of neighboring pixels may include at least two pixels with initial green component values ​​that are in the same row and adjacent to the pixel in the input image and at least two pixels with the same color component value as the pixel, for example, two pixels with initial green component values ​​on the left and right sides of the pixel, respectively, and two pixels with the same color component value as the pixel, respectively, on the left and right sides of the pixel.

[0063] The third group of neighboring pixels may include at least two pixels that are adjacent to the pixel in the input image and have an initial green component value. For example, the third group of neighboring pixels may include four pixels that are adjacent to the pixel and above, below, to the left, and to the right of the pixel.

[0064] For example, if pixel 0 is at a vertical edge or a horizontal edge of the input image, the initial green component value of pixel 0 may be determined based on the pixel values ​​of pixel 0 and four pixels adjacent to pixel 0. For example, the initial green component value of pixel 0 may be calculated as follows: initial green component value of pixel 0 = first average value - second average value + pixel value of pixel 0

[0065] Among them, the first average value = (initial green component value of pixel 1 + initial green component value of pixel 2) / 2; the second average value = (pixel value of pixel 3 + pixel value of pixel 0*2 + pixel value of pixel 4) / 4.

[0066] If pixel 0 is at a vertical edge of the input image, then pixel 1 and pixel 2 are the closest two pixels above and below pixel 0 with the initial green component value, and pixel 3 and pixel 4 are the closest two pixels above and below pixel 0, respectively, with the same color component value as pixel 0.

[0067] If pixel 0 is at a horizontal edge of the input image, then pixel 1 and pixel 2 are the closest two pixels with the initial green component value on the left and right of pixel 0, and pixel 3 and pixel 4 are the closest two pixels with the same color component value as pixel 0 on the left and right of pixel 0, respectively.

[0068] If pixel 0 is not at the vertical edge and horizontal edge of the input image, the initial green component value of pixel 0 can be calculated as follows: initial green component value of pixel 0 = (initial green component value of pixel 1 + initial green component value of pixel 2 + initial green component value of pixel 3 + initial green component value of pixel 4) / 4, where pixel 1, pixel 2, pixel 3 and pixel 4 are four pixels adjacent to pixel 0 and above, below, to the left and to the right of pixel 0, respectively.

[0069] The above-mentioned embodiments are mainly based on the derivative of the color difference method theory, that is, the color difference of the local area of ​​the image can be approximately kept unchanged. In this way, by considering whether the pixel is at the vertical edge or horizontal edge of the input image to select a corresponding group of adjacent pixels to perform green channel interpolation, the robustness of the interpolation result can be significantly improved, and the fluctuation caused by the jump of local noise can be tolerated.

[0070] The interpolation operation for the blue channel of the second group of pixels and the interpolation operation for the red channel of the third group of pixels are similar and will be described together below.

[0071] For pixels without initial green component values ​​and pixels with initial green component values ​​in the second group of pixels and the third group of pixels, interpolation operations for the blue channel and the red channel may be performed in different ways.

[0072] For the convenience of description, a first target pixel and a second target pixel are defined below.

[0073] In some embodiments, a first target pixel may be defined, which may be any pixel in the second set of pixels having an initial red component value in the input image, or may be any pixel in the third set of pixels having an initial blue component value in the input image.

[0074] The first diagonal gradient value can be determined based on the initial green component value of the first target pixel, the initial target component value of the first group of diagonal pixels, and the initial green component value. The second diagonal gradient value can be determined based on the initial green component value of the first target pixel, the initial target component value of the second group of diagonal pixels, and the initial green component value. It can be understood that the initial green component value mentioned here may be the initial green component value in the input image, or it may be the initial green component value obtained after performing an interpolation operation on the green channel, depending on which specific pixel is being targeted. For example, for the first target pixel, since it is any pixel with an initial red component value in the second group of pixels or any pixel with an initial blue component value in the third group of pixels, the initial green component value of the first target pixel is obtained after the above-mentioned interpolation operation on the green channel.

[0075] In addition, the first group of diagonal pixels and the second group of diagonal pixels are both relative to the first target pixel. For example, the first group of diagonal pixels and the second group of diagonal pixels can be located on two different diagonal lines where the first target pixel is located. For example, the first group of diagonal pixels may include two pixels at the upper left and lower right of the first target pixel, and the second group of diagonal pixels may include two pixels at the upper right and lower left of the first target pixel.

[0076] In addition, if the first target pixel belongs to the second group of pixels, since the blue channel interpolation is to be performed for the second group of pixels, in this case, the initial target component value may refer to the initial blue component value. If the first target pixel belongs to the third group of pixels, since the red channel interpolation is to be performed for the third group of pixels, in this case, the initial target component value may refer to the initial red component value.

[0077] Afterwards, based on the comparison result of the first diagonal gradient value and the second diagonal gradient value, the initial target component value of the first target pixel is determined based on the initial green component value of the first target pixel and based on the initial target component value of the first group of object pixels and / or the initial target component value of the second group of diagonal pixels.

[0078] For example, if the difference between the first diagonal gradient value and the second diagonal gradient value is greater than a preset diagonal gradient threshold, the initial target component value of the first target pixel can be determined based on the initial green component value of the first target pixel and the initial target component value and initial green component value of the second group of diagonal pixels.

[0079] If the difference between the second diagonal gradient value and the first diagonal gradient value is greater than the preset diagonal gradient threshold, the initial target component value of the first target pixel can be determined based on the initial green component value of the first target pixel and the initial target component value and initial green component value of the first group of diagonal pixels.

[0080] If the absolute difference between the first diagonal gradient value and the second diagonal gradient value is less than or equal to a preset diagonal gradient threshold, the initial target component value of the first target pixel can be determined based on the initial green component value of the first target pixel, the initial target component value and initial green component value of the first group of diagonal pixels, and the initial target component value and initial green component value of the second group of diagonal pixels.

[0081] In some embodiments, a second target pixel may be defined. The second target pixel may be any pixel in the second group of pixels that has an initial green component value in the input image, or may be any pixel in the third group of pixels that has an initial green component value in the input image. The initial target component value of the second target pixel may be determined based on the initial target component values ​​of a group of adjacent pixels.

[0082] The group of neighboring pixels may be pixels that are adjacent to the second target pixel and have an initial target component value in the input image. For example, if the second target pixel belongs to the second group of pixels, since blue channel interpolation is to be performed on the second group of pixels, the group of neighboring pixels may include pixels that are adjacent to the second target pixel (e.g., above and below the second target pixel) and have an initial blue component value in the input image.

[0083] In addition, if the second target pixel belongs to the second group of pixels, since the blue channel interpolation is to be performed on the second group of pixels, in this case, the initial target component value may refer to the initial blue component value. If the second target pixel belongs to the third group of pixels, since the red channel interpolation is to be performed on the third group of pixels, in this case, the initial target component value may refer to the initial red component value.

[0084] After performing the above interpolation operation, an initial green channel image, an initial blue channel image, and an initial red channel image can be obtained. However, in some cases, these initial images may be affected by noise or interpolation errors and have anomalies such as pseudo color and breakpoints. In order to avoid such influences and further improve the accuracy and reliability of each channel image, outlier correction can be performed. Outlier correction can be performed for each pixel. For example, in some embodiments, for each pixel in a plurality of pixels, the green component value, blue component value, and red component value of the pixel can be calculated based on the initial green component value, initial blue component value, and initial red component value of the pixel and a group of adjacent pixels of the pixel.

[0085] For example, the group of neighboring pixels may include eight pixels around the pixel, specifically, pixels above, below, to the left, to the right, to the upper left, to the lower left, to the upper right, and to the lower right of the pixel.

[0086] For example, in some implementations, the reference value of the pixel may be determined based on the initial green component value, the initial blue component value, and the initial red component value of the pixel. The reference value of the adjacent pixel may be determined for the initial green component value, the initial blue component value, and the initial red component value of each adjacent pixel in the group of adjacent pixels, so that a group of adjacent pixel reference values ​​may be obtained.

[0087] The first correction value of the pixel may be determined based on a comparison between the reference value of the pixel and a maximum value and a minimum value in the set of adjacent pixel reference values.

[0088] For example, if the reference value of the pixel is greater than the maximum value, the first correction value of the pixel can be determined as the maximum value; if the reference value of the pixel is less than the minimum value, the first correction value of the pixel can be determined as the minimum value; if the reference value of the pixel is less than or equal to the maximum value or greater than or equal to the minimum value, the first correction value of the pixel is determined as the reference value of the pixel.

[0089] The second corrected value of the pixel may be determined based on the initial green component value, the initial blue component value, and the initial red component value of the pixel. The third corrected value of the pixel may be determined based on the initial blue component value and the initial red component value of the pixel.

[0090] The green component value, blue component value, and red component value of the pixel may be calculated based on the first corrected value, the second corrected value, and the third corrected value of the pixel. For example, the green component value, blue component value, and red component value of the pixel may be calculated through matrix operations.

[0091] In this way, through outlier correction, problems such as false color and breakpoints in each channel image can be avoided.

[0092] In order to better understand the technical solution of the present disclosure, the following will be further described in conjunction with specific examples. It should be understood that the following examples do not impose any limitation on the scope of the present disclosure.

[0093] Figure 2 is a schematic flow chart of a process for demosaicing an image according to some embodiments.

[0094] like Figure 2 As shown, in stage 202, gradient calculation may be performed based on the input image 240 to generate a horizontal gradient map and a vertical gradient map.

[0095] The input image 240 may include a plurality of pixels and have a bayer format. An example of how to perform the gradient technique is described below.

[0096] Figure 3A An example of a portion of pixels of an input image 240 is shown.

[0097] exist Figure 3A In the example, it is assumed that the input image 240 is arranged in RGGB manner, where R represents pixels collecting initial red component values, G represents pixels collecting initial green component values, and B represents pixels collecting initial blue component values.

[0098] In addition, Figure 3A In the example of , the coordinates of each pixel are shown. For example, for a pixel with coordinates (pos_x, pos_y), pos_x can represent its position in the horizontal direction of the input image 240, and pos_y can represent its position in the vertical direction of the input image 240. The coordinates of other pixels are also represented in a similar manner. In the following description, the pixel will be referred to by the coordinates of the pixel, and p() will represent the pixel value. It can be understood that the pixel value of the pixel can be a pixel value in the input image, such as one of the initial red component value, the initial green component value, or the initial blue component value.

[0099] The following takes the pixel at the coordinates (pos_x, pos_y) as an example to illustrate how to calculate the horizontal gradient value and the vertical gradient value.

[0100] For the pixel at coordinate (pos_x, pos_y), the horizontal gradient value H_gradient can be calculated as follows:

[0101] H_gradient=abs(2*p(pos_x,pos_y)-p(pos_x,lef_2)-p(pos_x,rig_2))+abs(p(pos-x,lef_1)-p(p os_x,rig_1))+abs(p(pos_x,lef_3)-p(pos_x,lef_1)) / 2+abs(p(pos_x,rig_1)-p(pos_x,rig_3)) / 2

[0102] For the pixel at coordinate (pos_x, pos_y), the vertical gradient value V_gradient can be calculated as follows:

[0103] V_gradient=abs(2*p(pos_x,pos_y)-p(up_2,pos_y)-p(bot_2,pos_y))+abs(p(up_1,pos_y)-p(b ot_1,pos_y))+abs(p(up_3,pos_y)-p(up_1,pos_y)) / 2+abs(p(bot_3,pos_y)-p(bot_1,pos_y)) / 2

[0104] In the above equation, abs() may represent an absolute value; p(pos_x, pos_y) may represent a pixel value of a pixel at coordinates (pos_x, pos_y). Figure 3A As shown, the pixel at the coordinate (pos_x, pos_y) has an initial red component value in the input image 240, so p(pos_x, pos_y) can represent the initial red component value of the pixel at the coordinate (pos_x, pos_y). The meanings of other similar symbols are not repeated here.

[0105] The horizontal gradient value and the vertical gradient value can be calculated for pixels at each position in a similar manner as described above, thereby obtaining a horizontal gradient map and a vertical gradient map.

[0106] Return to Figure 2 In stage 204, the direction of each pixel can be determined based on the horizontal gradient map and the vertical gradient map, thereby generating a first direction map.

[0107] As described above, the direction of the pixel may include a horizontal direction, a vertical direction, or no direction. The horizontal direction may indicate that the pixel is at a horizontal edge of the input image 240. The vertical direction may indicate that the pixel is at a vertical edge of the input image 240. No direction may indicate that the pixel is not at a horizontal edge or a vertical edge of the input image 240.

[0108] For a certain pixel, if the absolute difference between the vertical gradient value and the horizontal gradient value of the pixel is greater than a preset gradient threshold, it can be indicated that the pixel is at a directional edge. Specifically, if the difference between the vertical gradient value of the pixel and the horizontal gradient value of the pixel is greater than the preset gradient threshold, it is determined that the pixel has a horizontal direction; if the difference between the horizontal gradient value of the pixel and the vertical gradient value of the pixel is greater than the preset gradient threshold, it is determined that the pixel has a vertical direction; if the absolute difference between the vertical gradient value of the pixel and the horizontal gradient value of the pixel is less than or equal to the preset gradient threshold, it is determined that the pixel has no direction.

[0109] For example, for the pixel at the above coordinates (pos_x, pos_y), if abs(V_gradient-H_gradient)>T, it can be indicated that the pixel is at a directional edge. Specifically, if V_gradient-H_gradient>T, it can be indicated that the pixel has a horizontal direction, that is, it is at a horizontal edge; if H_gradient-V_gradient>T, it can be indicated that the pixel has a vertical direction, that is, it is at a vertical edge; if abs(V_gradient-H_gradient)≤T, it can be indicated that the pixel has no direction, that is, it is not at a horizontal edge or a vertical edge. T can represent a preset gradient threshold.

[0110] In this way, after the direction of each pixel is determined, a first direction map can be obtained.

[0111] In stage 206, convolution filtering may be performed on the horizontal gradient values ​​and vertical gradient values ​​of the first group of non-directional pixels to obtain the horizontal filtered gradient values ​​and vertical filtered gradient values ​​of the first group of non-directional pixels. The first group of non-directional pixels may be pixels having no direction in the first direction map.

[0112] For example, each non-directional pixel can be used as the center, and the horizontal gradient value and the vertical gradient value can be convoluted using a convolution kernel. The size of the convolution kernel and the size of each element can be determined according to factors such as actual application scenarios and requirements.

[0113] Figure 3B An example of a convolution kernel is shown in Figure 3B In the example, the convolution kernel has a size of 3x3 and has Figure 3B The value of the element shown.

[0114] In stage 208, the direction of the first group of directionless pixels can be re-determined based on the horizontal filtering gradient value and the vertical filtering gradient value of the first group of directionless pixels, and the direction of the first group of directionless pixels can be replaced with the re-determined direction of the first group of directionless pixels in the first direction map, thereby obtaining a second direction map.

[0115] The process of determining the pixel direction based on the horizontal filtering gradient value and the vertical filtering gradient value is similar to the process of determining the pixel direction based on the horizontal gradient value and the vertical gradient value.

[0116] For example, for a pixel in the first group of directionless pixels, if the difference between the vertical filtering gradient value of the pixel and the horizontal filtering gradient value of the pixel is greater than a preset filtering gradient threshold, it is determined that the pixel has a horizontal direction; if the difference between the horizontal filtering gradient value of the pixel and the vertical filtering gradient value of the pixel is greater than the preset filtering gradient threshold, it is determined that the pixel has a vertical direction; if the absolute difference between the vertical filtering gradient value of the pixel and the horizontal filtering gradient value of the pixel is less than or equal to the preset filtering gradient threshold, it is determined that the pixel has no direction.

[0117] In stage 210, the direction of the second group of non-directional pixels can be re-determined based on the adjacent pixel information of the second group of non-directional pixels. The second group of non-directional pixels can be pixels with no direction in the second directional map. The direction of the second group of non-directional pixels can be replaced with the re-determined direction of the second group of non-directional pixels in the second directional map, thereby obtaining a usable directional map.

[0118] For example, for each pixel in the second group of directionless pixels: if the neighboring pixels above, below, left and right of the pixel have the same direction in the second direction map, the direction of the pixel is determined to be this direction; if the neighboring pixels above, upper left, upper right, below, lower left and lower right of the pixel have a horizontal direction in the second direction map, the direction of the pixel is determined to be a horizontal direction; if the neighboring pixels to the left, upper left, lower left, right, upper right and lower right of the pixel have a vertical direction, the direction of the pixel is determined to be a vertical direction.

[0119] For ease of understanding, Figure 3C An example of direction determination at stage 210 is shown.

[0120] exist Figure 3C In the example of FIG. 1 , if neighboring pixels 304 - 1 , 306 - 1 , 308 - 1 , and 310 - 1 of pixel 302 - 1 have the same direction in the second direction map, the direction of pixel 302 - 1 in the available direction map may be determined as the direction.

[0121] If the neighboring pixels 304-2, 312-2, 314-2, 306-2, 316-2, 318-2 of the pixel 302-2 have horizontal directions in the second direction map, the direction of the pixel 302-2 in the available direction map may be determined as a horizontal direction.

[0122] If the neighboring pixels 308-3, 312-3, 316-3, 310-3, 314-3, and 318-3 of the pixel 302-3 have vertical directions, the direction of the pixel 302-3 in the available direction map may be determined as a vertical direction.

[0123] In addition, Figure 3C 3 shows judgment matrices 350, 360 and 370 corresponding to the three judgment modes respectively.

[0124] Return to Figure 2 In stage 212, an interpolation operation may be performed on a green channel of a first group of pixels among the plurality of pixels based on the available direction map, thereby obtaining an initial green channel image.

[0125] The first group of pixels may be pixels that do not have an original green component value in the input image 240 , that is, pixels having an original red component value and pixels having an original blue component value are filled with the original green component value.

[0126] For ease of understanding, Figure 3D Another example of a portion of pixels of the input image 240 is shown. It can be understood that in order to make the example more targeted, Figure 3D The representation of the input image in Figure 3A The representation of the input image is slightly different. Figure 3D In the example of , the pixel values ​​of each pixel are shown, where R can represent the red component, G can represent the green component, and B can represent the blue component. For example, R33 can represent the initial red component value of the pixel at the corresponding position. G23 can represent the initial green component value of the pixel at the corresponding position. B22 can represent the initial blue component value of the pixel at the corresponding position.

[0127] The following takes the pixel corresponding to R33 as an example to illustrate how to perform green channel interpolation.

[0128] For the pixel corresponding to R33, if the pixel has a vertical direction in the available direction map, its initial green component value G33 can be calculated as follows:

[0129] G33=G_ave-R_ave+R33

[0130] Where, G_ave=(G23+G43) / 2, R_ave=(R13+2*R33+R53) / 4

[0131] If the pixel has a horizontal orientation in the available orientation map, its initial green component value G33 can be calculated as follows:

[0132] G33=G_ave-R_ave+R33

[0133] Where, G_ave=(G32+G34) / 2, R_ave=(R31+2*R33+R35) / 4

[0134] If the pixel has no orientation in the available orientation map, its initial green component value G33 can be calculated as follows:

[0135] G33=(G23+G43+G32+G34) / 4

[0136] For another example, for the pixel corresponding to B22, if the pixel has no direction in the available direction map, its initial green component value B22 = (G12 + G32 + G21 + G23) / 4

[0137] For pixels with initial blue component values ​​in the input image, green channel interpolation is similar and will not be described here.

[0138] In stage 214, interpolation operations may be performed on the blue channel of the second group of pixels in the plurality of pixels and on the red channel of the third group of pixels in the plurality of pixels, respectively, based on the initial green channel image, thereby obtaining an initial blue channel image and an initial red channel image. The second group of pixels may be pixels that do not have initial blue component values ​​in the input image, and the third group of pixels may be pixels that do not have initial red component values ​​in the input image.

[0139] Below Figure 3D Take the pixel corresponding to R33 in as an example to illustrate how to perform blue channel interpolation.

[0140] First, the first diagonal gradient value N_grad and the second diagonal gradient value P_grad of the pixel can be calculated as follows:

[0141] N_grad=abs(B22-B44)+abs(2*G33-G22-G44)

[0142] P_grad=abs(B24-B42)+abs(2*G33-G24-G42)

[0143] Among them, G33, G22, G44, G24, G42 may be initial green component values ​​obtained by performing green channel interpolation on pixels at corresponding positions in stage 212. For example, G22 is an initial green component value obtained by performing green channel interpolation on pixels corresponding to B22.

[0144] The initial blue component value B33 of the pixel corresponding to R33 can be calculated as follows:

[0145] If P_grad-N_grad>T2, then B33=(B22+B44) / 2+(2*G33-G22-G44);

[0146] If N_grad-P_grad>T2, then B33=(B24+B42) / 2+(2*G33-G24-G42);

[0147] If abs(P_grad-N_grad)≤T2, then B33=(B22+B24+B42+B44) / 4+(4*G33-G22-G24-G42-G44) / 8

[0148] Wherein, T2 may represent a preset diagonal gradient threshold.

[0149] For pixels having an initial green component value in the input image 240, blue channel interpolation may be performed in different ways.

[0150] For example, for Figure 3D For the pixel corresponding to G32 in , its initial blue component value B32 can be calculated as follows:

[0151] B32=(B22+B42) / 2.

[0152] For example, for Figure 3D For the pixel corresponding to G23 in , its initial blue component value B23 can be calculated as follows:

[0153] B23=(B22+B24) / 2.

[0154] The interpolation process for the red channel is similar to the interpolation process for the blue channel, and will not be described in detail here.

[0155] In this way, through the above interpolation process, an initial green channel image, an initial blue channel image and an initial red channel image can be obtained.

[0156] return Figure 2 In stage 216, outlier correction may be performed based on the initial green channel image, the initial blue channel image, and the initial red channel image to obtain a green channel image, a blue channel image, and a red channel image.

[0157] The following will use specific examples to illustrate how to correct outliers. Figure 3E An example of nine pixels is shown. Figure 3E In the example of , it is assumed that outlier correction is performed on pixel pc.

[0158] Can be targeted Figure 3EFor all pixels p1 to p8 and pc in , their respective reference values ​​are calculated. For example, for each of these pixels, its reference value can be calculated based on its initial blue component value, initial red component value and initial green component value, such as the reference value can be calculated as (B+R+2*G) / 4, where B represents the initial blue component value of the corresponding pixel, R represents the initial red component value of the corresponding pixel, and G represents the initial green component value.

[0159] Assume that Max_value represents the maximum value of the eight reference values ​​of pixels p1 to p8, Min_value represents the minimum value of the eight reference values ​​of pixels p1 to p8, Y1_Pc represents the reference value of pixel pc, Y1 represents the first correction value of pixel pc, Y2 represents the second correction value of pixel pc, and Y3 represents the third correction value of pixel pc.

[0160] If Y1_Pc is greater than Max_value, it can be determined that Y1=Max_value; if Y1_Pc is less than Min_value, it can be determined that Y1=Min_value; otherwise, Y1=Y1_Pc.

[0161] In addition, it can be determined that Y2=(B1_pc+R1_pc+2*G1_pc) / 4, Y3=(B1_pc-R1_pc) / 2, where B1_pc can represent the initial blue component value of the pixel pc, R1_pc can represent the initial red component value of the pixel pc, and G1_pc can represent the initial green component value of the pixel pc.

[0162] Then, based on Y1, Y2 and Y3, the blue component value, the red component value and the green component value of the pixel pc can be calculated, for example, this can be calculated by matrix operation.

[0163] For example, the blue component value B_pc, the red component value R_pc, and the green component value G_pc of the pixel pc may be calculated in the following manner:

[0164]

[0165] The correction of abnormal points for other pixels is similar and will not be described here.

[0166] After outlier correction is performed, a green channel image 250 , a blue channel image 260 , and a red channel image 270 may be obtained.

[0167] It can be seen that through the embodiments of the present disclosure, image de-mosaicing can be implemented with low computational complexity and low cost, and good performance can be achieved.

[0168] Figure 4is a schematic block diagram of an apparatus for demosaicing an image according to some embodiments.

[0169] like Figure 4 As shown, the apparatus 400 may include a gradient determining unit 402 , a direction determining unit 404 , and an interpolation unit 406 .

[0170] The gradient determination unit 402 may determine gradient data based on an input image. The input image may include a plurality of pixels and have a Bayer format. The gradient data may include a horizontal gradient value of each pixel in the plurality of pixels along the horizontal direction of the input image and a vertical gradient value along the vertical direction of the input image.

[0171] The direction determination unit 404 may determine the direction data based on the gradient data. The direction data may be used to indicate whether each pixel in the plurality of pixels is at a horizontal edge or a vertical edge of the input image.

[0172] The interpolation unit 406 may perform an interpolation operation on the input image based on the direction data to obtain a set of output images corresponding to the input image. The set of output images may include a green channel image, a blue channel image, and a red channel image of multiple pixels.

[0173] Each unit of the device 400 can execute the specific process described above with respect to the method embodiment. Therefore, for the sake of brevity of description, the specific operations and functions of each unit of the device 400 are not described in detail here.

[0174] Figure 5 is a schematic block diagram of an apparatus for demosaicing an image according to some embodiments.

[0175] like Figure 5 As shown, the apparatus 500 may include a processor 502, a memory 504, an input interface 506, and an output interface 508, and these modules may be coupled together via a bus 510. However, it should be understood that Figure 5 This is only an example and does not limit the scope of the present disclosure. For example, in different application scenarios, the device 500 may include more or fewer modules, which is not limited herein.

[0176] The memory 504 may be used to store various information related to the functions or operations of the device 500 (such as the input image, gradient data, direction data, output image, etc. mentioned above), executable instructions or codes, etc. For example, the memory 504 may include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a flash memory, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a register, a hard disk, and the like.

[0177] The processor 502 may be used to execute or implement various functions or operations of the apparatus 500, such as the various operations described herein. For example, the processor 502 may execute executable instructions or codes stored in the memory 504, thereby implementing the various processes described above with respect to the method embodiments. The processor 502 may include various applicable processors, for example, a general-purpose processor (such as a central processing unit (CPU)), a special-purpose processor (such as a digital signal processor, a special-purpose integrated circuit, etc.).

[0178] The input interface 506 can receive data in various forms, such as the input image described above. The output interface 508 can output data in various forms, such as the output image described above.

[0179] The embodiments of the present disclosure further provide a computer-readable storage medium. The computer-readable storage medium may store executable codes, and when the executable codes are executed, the specific processes described above with respect to the method embodiments are implemented.

[0180] For example, computer-readable storage media may include, but are not limited to, RAM, ROM, Electrically-Erasable Programmable Read-Only Memory (EEPROM), Static Random Access Memory (SRAM), hard disk, flash memory, and the like.

[0181] Specific embodiments of the present disclosure are described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0182] Not all steps and units in the above processes and system structure diagrams are necessary, and some steps or units may be omitted according to actual needs. The device structure described in the above embodiments may be a physical structure or a logical structure, that is, some units may be implemented by the same physical entity, some units may be implemented by multiple physical entities, or may be implemented by some components in multiple independent devices.

[0183] The optional implementation modes of the embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present disclosure are not limited to the specific details in the above implementation modes. Within the technical concept of the embodiments of the present disclosure, various modifications can be made to the technical solutions of the embodiments of the present disclosure, and these modifications all belong to the protection scope of the embodiments of the present disclosure.

Claims

1. A method for demosaicing an image, include: Based on an input image, determining gradient data, wherein the input image includes a plurality of pixels and has a Bayer format, and the gradient data includes a horizontal gradient value of each pixel of the plurality of pixels along a horizontal direction of the input image and a vertical gradient value along a vertical direction of the input image; Determining direction data based on the gradient data, wherein the direction data is used to indicate whether each pixel of the plurality of pixels is at a horizontal edge or a vertical edge of the input image; Based on the direction data, an interpolation operation is performed on the input image to obtain a set of output images corresponding to the input image, wherein the set of output images includes a green channel image, a blue channel image, and a red channel image of the plurality of pixels.

2. The method according to claim 1, in, Determining the gradient data includes: For each pixel in the plurality of pixels: Determine a horizontal gradient value of the pixel based on a pixel value of the pixel in the input image and pixel values ​​of a group of pixels in the same row as the pixel in the input image; A vertical gradient value of the pixel is determined based on a pixel value of the pixel in the input image and pixel values ​​of a group of pixels in the same column as the pixel in the input image.

3. The method according to claim 1, in, The direction data includes an available direction map for the plurality of pixels, the available direction map is used to indicate the direction of each pixel in the plurality of pixels, the direction of each pixel is a horizontal direction, a vertical direction or no direction, the horizontal direction indicates being at the horizontal edge, the vertical direction indicates being at the vertical edge, and the no direction indicates not being at the horizontal edge and the vertical edge; Determining the direction data includes: Based on the gradient data, generating a first direction map for the plurality of pixels, wherein the first direction map is used to represent the direction of each pixel determined based on the gradient data; Based on the first direction map, determining a first group of non-directional pixels, wherein the first group of non-directional pixels are pixels among the plurality of pixels that have no direction in the first direction map; Based on the neighboring pixel information of the first group of non-directional pixels, the directions of the first group of non-directional pixels are re-determined to generate the available direction map.

4. The method according to claim 3, in, Generating the first directional diagram includes: For each pixel in the plurality of pixels: If the difference between the vertical gradient value of the pixel and the horizontal gradient value of the pixel is greater than a preset gradient threshold, it is determined that the pixel has a horizontal direction; If the difference between the horizontal gradient value of the pixel and the vertical gradient value of the pixel is greater than the preset gradient threshold, it is determined that the pixel has a vertical direction; If the absolute difference between the vertical gradient value of the pixel and the horizontal gradient value of the pixel is less than or equal to the preset gradient threshold, it is determined that the pixel has no direction.

5. The method according to claim 3, in, Based on the adjacent pixel information of the first set of non-oriented pixels, re-determine the directions of the first set of non-oriented pixels to generate the available direction map, including: Based on the horizontal gradient values and vertical gradient values of the adjacent pixels of the first set of non-oriented pixels, filter the horizontal gradient values and vertical gradient values of the first set of non-oriented pixels respectively to determine the horizontal filtered gradient values and vertical filtered gradient values of the first set of non-oriented pixels; Based on the horizontal filtered gradient values and vertical filtered gradient values of the first set of non-oriented pixels, re-determine the directions of the first set of non-oriented pixels so as to obtain a second direction map from the first direction map; Based on the second direction map, determine a second set of non-oriented pixels, where the second set of non-oriented pixels are the pixels among the multiple pixels that are non-oriented in the second direction map; Based on the adjacent pixel information of the second set of non-oriented pixels, re-determine the directions of the second set of non-oriented pixels so as to obtain the available direction map from the second direction map.

6. The method according to claim 5, wherein, Filtering the horizontal gradient values and vertical gradient values of the first set of non-oriented pixels respectively includes: For each pixel in the first set of non-oriented pixels, perform convolution filtering on the horizontal gradient value and vertical gradient value of this pixel respectively.

7. The method according to claim 5, wherein, Re-determining the directions of the first set of non-oriented pixels based on the horizontal filtered gradient values and vertical filtered gradient values of the first set of non-oriented pixels includes: For each pixel in the first set of non-oriented pixels: If the difference between the vertical filtered gradient value of this pixel and the horizontal filtered gradient value of this pixel is greater than a preset filtered gradient threshold, determine that this pixel has a horizontal direction; If the difference between the horizontal filtered gradient value of this pixel and the vertical filtered gradient value of this pixel is greater than the preset filtered gradient threshold, determine that this pixel has a vertical direction; If the absolute difference between the vertical filtered gradient value of this pixel and the horizontal filtered gradient value of this pixel is less than or equal to the preset filtered gradient threshold, determine that this pixel is non-oriented.

8. The method according to claim 5, wherein, Re-determining the directions of the second set of non-oriented pixels based on the adjacent pixel information of the second set of non-oriented pixels includes: For each pixel in the second set of non-oriented pixels, if the directions of a set of adjacent pixels of this pixel in the second direction map are the same, determine the direction of the set of adjacent pixels in the second direction map as the direction of this pixel in the available direction map.

9. The method according to claim 8, wherein, The set of adjacent pixels includes the pixels above, below, to the left, and to the right of this pixel; The set of adjacent pixels includes the pixels above, to the upper left, and to the upper right of this pixel and the pixels below, to the lower left, and to the lower right of this pixel, and the set of adjacent pixels has a horizontal direction in the second direction map; or The group of adjacent pixels includes pixels on the left, upper left, and lower left of the pixel and pixels on the right, upper right, and lower right of the pixel, and the group of adjacent pixels has a vertical direction in the second direction map.

10. The method according to claim 1, in, Based on the direction data, an interpolation operation is performed on the input image to obtain a set of output images corresponding to the input image, including: Based on the direction data, an interpolation operation is performed on the input image to generate an initial green channel image, an initial blue channel image, and an initial red channel image of the plurality of pixels, wherein the initial green channel image includes initial green component values ​​of the plurality of pixels, the initial blue channel image includes initial blue component values ​​of the plurality of pixels, and the initial red channel image includes initial red component values ​​of the plurality of pixels; Outlier correction is performed based on the initial green channel image, the initial blue channel image, and the initial red channel image to generate the green channel image, the blue channel image, and the red channel image, wherein the green channel image includes green component values ​​of the multiple pixels, the blue channel image includes blue component values ​​of the multiple pixels, and the red channel image includes red component values ​​of the multiple pixels.

11. The method according to claim 10, in, In the input image, an initial pixel value of each pixel in the plurality of pixels is one of an initial red component value, an initial blue component value, or an initial green component value; Based on the direction data, performing an interpolation operation on the input image includes: Based on the direction data, performing an interpolation operation on a green channel of a first group of pixels among the plurality of pixels to generate the initial green channel image, wherein the first group of pixels does not have an initial green component value in the input image; Based on the initial green channel image, interpolation operations are performed on the blue channel of a second group of pixels among the multiple pixels and on the red channel of a third group of pixels among the multiple pixels, respectively, to generate the initial blue channel image and the initial red channel image, wherein the second group of pixels does not have an initial blue component value in the input image, and the third group of pixels does not have an initial red component value in the input image.

12. The method according to claim 11, in, Performing an interpolation operation on a green channel of a first group of pixels among the plurality of pixels comprises: For each pixel in the first group of pixels: If the pixel is at the vertical edge, determining an initial green component value of the pixel based on the pixel value of the pixel and the pixel values ​​of a first group of neighboring pixels in the same column as the pixel; If the pixel is at the horizontal edge, determining an initial green component value of the pixel based on the pixel value of the pixel and the pixel values ​​of a second group of neighboring pixels in the same row as the pixel; If the pixel is not at the vertical edge and the horizontal edge, an initial green component value of the pixel is determined based on pixel values ​​of a third group of neighboring pixels around the pixel.

13. The method according to claim 12, in, The first group of neighboring pixels includes at least two pixels having an initial green component value and at least two pixels having the same color component value as the pixel and being adjacent to the pixel in the same column; The second group of neighboring pixels includes at least two pixels having an initial green component value and at least two pixels having the same color component value as the pixel and being adjacent to the pixel in the same row; The third group of neighboring pixels includes at least two pixels that are immediately adjacent to the pixel and have initial green component values.

14. The method according to claim 11, in, Performing an interpolation operation on a blue channel of the second group of pixels and on a red channel of the third group of pixels, comprising: For a first target pixel, wherein the first target pixel is any pixel in the second group of pixels having an initial red component value or any pixel in the third group of pixels having an initial blue component value: determining a first diagonal gradient value based on an initial green component value of the first target pixel, initial target component values ​​and initial green component values ​​of a first group of diagonal pixels; determining a second diagonal gradient value based on an initial green component value of the first target pixel, initial target component values ​​and initial green component values ​​of a second set of diagonal pixels; determining an initial target component value of the first target pixel based on an initial green component value of the first target pixel and based on initial target component values ​​of the first group of diagonal pixels and / or initial target component values ​​of the second group of diagonal pixels according to a comparison result of the first diagonal gradient value and the second diagonal gradient value, in: The first group of diagonal pixels and the second group of diagonal pixels are respectively located on two different diagonal lines where the first target pixel is located. If the first target pixel belongs to the second group of pixels, the initial target component value is an initial blue component value, If the first target pixel belongs to the third group of pixels, the initial target component value is an initial red component value.

15. The method according to claim 14, in, Determining a target component value of the first target pixel includes: If the difference between the first diagonal gradient value and the second diagonal gradient value is greater than a preset diagonal gradient threshold, determining an initial target component value of the first target pixel based on the initial green component value of the first target pixel and the initial target component value and the initial green component value of the second group of diagonal pixels; If the difference between the second diagonal gradient value and the first diagonal gradient value is greater than the preset diagonal gradient threshold, determining an initial target component value of the first target pixel based on the initial green component value of the first target pixel and the initial target component value and the initial green component value of the first group of diagonal pixels; If the absolute difference between the first diagonal gradient value and the second diagonal gradient value is less than or equal to the preset diagonal gradient threshold, the initial target component value of the first target pixel is determined based on the initial green component value of the first target pixel, the initial target component value and initial green component value of the first group of diagonal pixels, and the initial target component value and initial green component value of the second group of diagonal pixels.

16. The method according to claim 11, in, Performing an interpolation operation on a blue channel of the second group of pixels and on a red channel of the third group of pixels, comprising: For a second target pixel, wherein the second target pixel is any pixel in the second group of pixels having an initial green component value or any pixel in the third group of pixels having an initial green component value: An initial target component value of the second target pixel is determined based on initial target component values ​​of a group of adjacent pixels, wherein: The set of neighboring pixels are pixels that are immediately adjacent to the second target pixel and have the initial target component value in the input image; If the second target pixel belongs to the second group of pixels, the initial target component value is an initial blue component value, If the second target pixel belongs to the third group of pixels, the initial target component value is an initial red component value.

17. The method according to claim 10, in, Performing outlier correction based on the initial green channel image, the initial blue channel image, and the initial red channel image includes: For each pixel in the plurality of pixels, a green component value, a blue component value, and a red component value of the pixel are calculated based on initial green component values, initial blue component values, and initial red component values ​​of the pixel and a group of neighboring pixels of the pixel.

18. The method according to claim 17, in, Calculating a green component value, a blue component value, and a red component value of the pixel based on an initial green component value, an initial blue component value, and an initial red component value of the pixel and a group of adjacent pixels of the pixel, comprising: Determine a reference value of the pixel based on an initial green component value, an initial blue component value, and an initial red component value of the pixel; For an initial green component value, an initial blue component value, and an initial red component value of each adjacent pixel in the group of adjacent pixels, determining a reference value of the adjacent pixel to obtain a group of adjacent pixel reference values; Determining a first corrected value of the pixel based on a comparison between the reference value of the pixel and a maximum value and a minimum value in the set of adjacent pixel reference values; Determining a second corrected value of the pixel based on an initial green component value, an initial blue component value, and an initial red component value of the pixel; Determining a third corrected value of the pixel based on an initial blue component value and an initial red component value of the pixel; Based on the first corrected value, the second corrected value and the third corrected value of the pixel, a green component value, a blue component value and a red component value of the pixel are calculated.

19. The method according to claim 18, in, Determining a first correction value of the pixel based on a comparison between the reference value of the pixel and a maximum value and a minimum value in the set of adjacent pixel reference values ​​comprises: If the reference value of the pixel is greater than the maximum value, determining the first corrected value of the pixel as the maximum value; If the reference value of the pixel is less than the minimum value, determining the first corrected value of the pixel as the minimum value; If the reference value of the pixel is less than or equal to the maximum value or greater than or equal to the minimum value, the first correction value of the pixel is determined as the reference value of the pixel.

20. A device for demosaicing an image, include: a gradient determination unit configured to: determine gradient data based on an input image, wherein the input image includes a plurality of pixels and has a Bayer format, and the gradient data includes a horizontal gradient value of each pixel of the plurality of pixels along a horizontal direction of the input image and a vertical gradient value along a vertical direction of the input image; a direction determination unit, configured to: determine direction data based on the gradient data, wherein the direction data is used to indicate whether each pixel of the plurality of pixels is at a horizontal edge or a vertical edge of the input image; The interpolation unit is configured to: perform an interpolation operation on the input image based on the direction data to obtain a set of output images corresponding to the input image, wherein the set of output images includes a green channel image, a blue channel image, and a red channel image of the multiple pixels.

21. A device for demosaicing an image, include: at least one processor; A memory in communication with the at least one processor, having executable codes stored thereon, which, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 19.

22. A computer-readable storage medium storing executable codes, wherein the executable codes implement the method according to any one of claims 1 to 19 when executed.

Images