Method and system for processing images

Abstract

Embodiments of the present invention generally relate to methods and systems for processing images. Specifically, an image processing method and system are described in which a filtering operation is applied to an image, wherein the filtering operation uses a filtering cost based on an image gradient in the image. In this way, image data for an image region is filtered depending on the image gradient for the image region. This can be useful for different situations, such as when images are combined to form a high dynamic range (HDR) image. The filtering operation can be used as part of a connectivity unit that determines a connected image region, and / or the filtering operation can be used as part of a fusion unit that fuses two or more images together to form a fused image.

Description

[0001] Related Applications

[0002] This application is a divisional application of the invention patent application with application number 201610159268.9, application date March 18, 2016, and invention name “Method and system for processing images”. Technical Field

[0003] Embodiments of the present invention generally relate to image processing, and more particularly to image gradient-based image filtering. Background Art

[0004] Cameras are used to capture images. One of the fundamental limitations of light sensors in digital cameras is their dynamic range, which describes the ratio between the minimum light intensity and the maximum light intensity that can be detected by the light sensor. Although quantization plays an important role in the achievable dynamic range, the limitations are often mainly physical in nature. The minimum detectable light intensity depends on the sensor element size, efficiency, and noise floor. Individual pixels are often very small in compact devices, resulting in a low probability of capturing photons in low light conditions and a high susceptibility to interference from various noise sources and neighboring pixels. The maximum detectable light intensity can be determined by the maximum amount of charge that the sensor element can generate based on light, which can lead to saturation effects. The exposure level of the image can be adjusted, for example, by adjusting the exposure time, the aperture size, or the sensor sensitivity. The exposure level of the image can be adjusted to suit the lighting conditions of the scene to be captured, for example, for dark scenes, the exposure level can be increased, and for bright scenes, the exposure level can be reduced. Adjusting the exposure level between images allows a wider operating range, but does not affect the dynamic range of a single exposure or image.

[0005] A high dynamic range (HDR) image can be constructed by merging (i.e., fusing) multiple images having different exposure levels, such as different exposure times and / or sensor sensitivity settings. For example, two or more images of a scene may be captured sequentially, and then the sequentially captured images may be fused to form an HDR image. For example, two images may be captured: a first image having a high exposure level (which may be referred to herein as a "long exposure"), and a second image having a low exposure level (which may be referred to herein as a "short exposure"). The two images can be combined such that in dark image areas (e.g., in shadows) the long exposure is primarily (e.g., only) used to form the fused HDR image, because the long exposure is likely to have less noise than the short exposure. However, in bright image areas (e.g., the sky), the long exposure may be saturated and thus the short exposure is primarily (e.g., only) used to form the fused HDR image to avoid unwanted saturation effects from the long exposure in these images. If the transitions between the images used to form the HDR image are sharp, they may be noticeable in the HDR image, which may be perceptually disadvantageous. Therefore, smoothing the transition over a series of pixels makes the transition not so sharp, and therefore not so noticeable in the HDR image. Even for the fastest sensors, there may be a noticeable delay between the images captured sequentially. The delay between images can be considered "noticeable" in the case where it is long enough to change in the image due to the time difference between the moments when the images are captured, for example due to motion within the scene or motion of the camera (for example, in the case where the camera is implemented in a handheld device). For many reasons, it can be proved that it is challenging to fuse two images in the case where there is motion between the images. For example, if an object moves between the two images that are then fused, artifacts such as ghosting effects can appear in the HDR image due to the difference in the position of the object in the two images. Motion in the scene due to dynamic objects or parallax can conceivably be tracked, but this would require complex processes that would typically require too much time and processing power to be implemented in real time on a mobile device, such as a smart phone, tablet, smart watch, or other suitable mobile device on which HDR processing can be performed and where processing power can be limited. It would be useful to have a method of handling motion when combining images to form an HDR image that is simple enough to implement in real time on a mobile device. Summary of the invention

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0007] A method of image processing is provided, comprising: determining an image gradient indication for at least one image; determining a filtering cost for an image region based on the determined image gradient indication for the at least one image; and processing data associated with the at least one image using the determined filtering cost for the image region, including applying a filtering operation such that processing for the image region is performed dependent on the image gradient of the at least one image.

[0008] The method may also include: determining a type mask, the type mask including type indicators for image regions to thereby classify each of the image regions into one of a plurality of types; wherein the processed data associated with at least one image may include, for each of a plurality of image regions: using a filtering operation to determine a minimum cumulative cost along a path from the image region to an image region of a specific type according to the type indicator; and selectively setting the type indicator for the image region to indicate a specific type depending on a comparison of the determined minimum cumulative cost with a threshold.

[0009] The at least one image may include multiple images, wherein at least one initial blending mask may indicate an initial blending factor for use when combining two images, and wherein processing data related to the at least one image may include smoothing the corresponding transition at a smoothing rate based on the filter cost for the image area at the transition in the at least one initial blending mask.

[0010] The processing of data associated with the at least one image includes applying denoising to the at least one image using a filtering operation such that an extent of the denoising applied to a region of the image depends on a filtering cost for the region of the image.

[0011] The at least one image may include a plurality of images captured with different focuses.

[0012] Processing data associated with at least one image includes applying denoising to the at least one image using a filtering operation such that an extent of the denoising applied to a region of the image depends on a filtering cost for the region of the image.

[0013] An image processing system is provided, comprising: gradient determination logic configured to determine an image gradient indication for at least one image; filtering cost determination logic configured to determine a filtering cost for an image region based on the determined image gradient indication for the at least one image; and processing logic configured to process data associated with the at least one image using the determined filtering cost for the image region, the processing logic comprising filtering logic configured to apply a filtering operation such that the processing logic is configured to perform processing for the image region in dependence on the image gradient of the at least one image.

[0014] A computer readable code may be provided, which is suitable for performing any of the steps in the method described herein when the code is run on a computer. In addition, the computer readable code may be provided for generating an image processing system according to any of the examples described herein. The computer readable code may be encoded on a computer readable storage medium.

[0015] It will be readily appreciated by those skilled in the art that the above features may be combined where appropriate and may be combined with any of the aspects of the examples described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Examples will now be described in detail with reference to the accompanying drawings, in which:

[0017] Figure 1 is a high-level diagram of a system for processing images;

[0018] Figure 2 is a functional block diagram of an HDR processing unit;

[0019] Figure 3 is a flow chart of a method for performing HDR processing on an image;

[0020] Figure 4 is a flow chart showing a detailed view of the filtering steps;

[0021] Figure 5 Represents the type mask for the image area;

[0022] Figure 6 A first example of processing a type mask using a connectivity filter is shown;

[0023] Figure 7 A second example of processing a type mask using a connectivity filter is shown;

[0024] Figure 8 illustrates the process of smoothing the blending mask; and

[0025] Fig. 9 is a schematic diagram of a computer system in which an HDR processing unit is implemented.

[0026] The accompanying drawings illustrate various examples. The skilled person will recognize that the illustrated element boundaries (e.g., boxes, groups of boxes, or other shapes) in the accompanying drawings represent an example of boundaries. Possibly, in some examples, an element may be designed as multiple elements or multiple elements may be designed as one element. Where appropriate, common reference numerals are used throughout the accompanying drawings to indicate similar features. DETAILED DESCRIPTION

[0027] The problem of handling motion when combining images to form an HDR image is mentioned above. As mentioned above, if the transitions between images used to form the HDR image are sharp, they may be noticeable in the HDR image, which may be perceptually disadvantageous. To solve this problem, the transitions can be smoothed over a series of pixels so that the transitions are not so sharp. The inventors have recognized that in order to avoid creating visible (i.e., noticeable) seams in the combined image, the transitions between the source images are preferably roughly as smooth as the local image texture. That is, the rate at which the transition occurs is preferably related to the image gradient of the image area over which the transition occurs. For example, in smooth image areas (i.e., image areas with small image gradients) such as image areas showing the sky or a painted surface such as a wall, the transitions between the images used to form the fused image preferably appear gradually over a relatively large distance; however, in detailed image areas (i.e., image areas with large image gradients) such as image areas showing text or a group of people or a series of small commodities, the transitions between the images used to form the fused image can appear relatively quickly over a relatively short distance without creating such noticeable seams in the fused image. If a transition would need to occur at a rate that would be noticeable between two image regions of different types, the image regions may be connected so that a transition is not performed between those two image regions.

[0028] Therefore, in the examples described herein, the image gradient for the image to be combined is determined. A filtering operation can be performed that uses a filtering cost based on the image gradient so as to filter image data related to the image region of the image depending on the image gradient for the image region. For example, as described in more detail below, the filtering operation can be used as part of a connectivity unit that determines connected image regions, and / or the filtering operation can be used as part of a fusion unit that fuses two or more images together to form a fused image. In other examples, the filtering operation can be used for other types of filtering that can appropriately depend on the image gradient in the image. For example, the filtering operation can be used to apply denoising to the image so that the extent of the denoising applied to the image region depends on the image gradient in the image region.

[0029] Embodiments will now be described by way of example in the context of frame-sequential high dynamic range image synthesis, which is the process of synthesizing a series of images taken in rapid succession with varying exposure times or sensor sensitivities (e.g., from a handheld device) in order to reconstruct an image of a scene with a higher dynamic range than a camera can capture in one exposure. Corresponding methods can be applied in other examples, for example, for combining multiple images captured with different focuses, where some image areas may be in better focus in one image and other image areas may be in better focus in another image, and the images are fused for the purpose of merging the image areas in best focus from the multiple images without introducing noticeable transitions between the images.

[0030] Figure 11 is a high-level diagram of a system for processing an image, wherein HDR processing is applied to the image. The system includes a sensor array 102, an image processing block 104, an HDR processing unit 106, and a memory 108. The system may be implemented, for example, in a device such as a camera, a smart phone, a tablet, a laptop, a smart watch, or any other suitable device. The sensor array 102 includes a plurality of light sensors for capturing raw image data. The raw image data may, for example, indicate the light intensity at a particular pixel to thereby represent an image. The raw image data is passed to the image processing block 104, which performs image processing on the raw image data to thereby determine the pixel values ​​of the image. The image processing operations performed by the image processing block 104 may be standard operations, such as white balance, denoising, defective pixel repair, gamut mapping, color enhancement, and the like. The image processing block 104 may be implemented in hardware or software, or a combination thereof. For example, the image processing block 104 can be implemented in hardware as a camera pipeline that receives raw image data (e.g., image data for pixels in a row-by-row order) from the sensor array 102 in a raster scan order. At a certain point in the processing pipeline of the image processing block 104, the image data for the image can be passed to the HDR processing unit 106. The HDR processing unit 106 performs HDR processing and provides the HDR image back to the image processing block 104, which can perform further processing on the HDR image. The processed image can then be output from the image processing block 104 and can be stored in the memory 108. The processed image can be used for other purposes, for example, the processed image can be passed to a display for display thereon, or can be sent to another device, etc. In other examples, the HDR processing unit 106 can be implemented as part of the image processing block 104. Additionally, in other examples, HDR processing may be the last processing performed on the image data, such that the enhanced HDR image from HDR processing unit 106 may not be passed back to image processing block 104 and may instead be used as an output such that it is, for example, provided to memory 108 or to a display for display thereon or to a transmission module for transmission to another device.

[0031] Figure 2A more detailed view of the HDR processing unit 106 is provided. The HDR processing unit 106 includes image registration logic 202, gradient determination logic 204, saturation determination logic 206, motion determination logic 208, filter cost determination logic 210, type mask determination logic 212, and connectivity unit 214 including connectivity filtering logic 216 and union-find logic 218. The HDR processing unit 106 also includes a fusion unit 220 including initial fusion mask determination logic 222, fusion filtering logic 224, image fusion mask determination logic 226, and image fusion logic 228. The connectivity unit 214 and the fusion unit 220 may be collectively referred to herein as processing logic 230. The components of the HDR processing unit 106 (e.g., Figure 2 The logical blocks and units shown in the figure are intended to correspond to many functional blocks. This is for illustration purposes only. Figure 2 It is not intended to define a strict division between different parts of the hardware on the chip or between different programs, processes or functions in the software. In some embodiments, some or all of the algorithms described herein may be performed in whole or in part in hardware. In some embodiments, at least a portion of the HDR processing unit 106 may be implemented by a processor (e.g., a graphics processing unit (GPU) or a central processing unit (CPU)) that functions under software control. Any such software is preferably stored on a non-transitory computer-readable medium, such as a memory (RAM, cache, hard disk, etc.) or other storage device (USB stick, CD, disk, etc.).

[0032] refer to Figure 3 and Figure 4 The flowchart in describes the operation of the HDR processing unit 106. The HDR method described herein handles situations where motion occurs between two images in a much simpler way than attempting to track motion, making the method described herein suitable for use on computer systems such as mobile devices where processing resources are greatly limited. Motion is handled by identifying image regions where motion occurs (e.g., based on the differences between the images) and treating these image regions as "inconsistent" regions where fusion should not occur, so in these inconsistent regions, the HDR image is formed from only one of the images. This avoids artifacts caused by motion such as ghosting effects.

[0033] In step S302, the HDR processing unit 106 receives a plurality of images, for example, from the image processing block 104. Figure 2 The images are received at the image registration logic 202 shown in FIG. The images have different exposure levels, so that the HDR processing unit 106 can combine the images with different exposure levels to thereby form an HDR image. For example, Figure 2The two images (I L and I S ), where image I L With high exposure level and image I S Having a low exposure level. Images with a high exposure level may be referred to herein as “long exposure” images, but the high exposure level may be attributed to one or more of many different factors such as a long exposure time, a large aperture size, and / or a high sensitivity setting of the sensors in the sensor array 102. Similarly, images with a low exposure level may be referred to herein as “short exposure” images, but the low exposure level may be attributed to one or more of many different factors such as a short exposure time, a small aperture size, and / or a low sensitivity setting of the sensors in the sensor array 102. Note that the order in which the images are captured is not important, i.e., the long exposure image may be captured before or after the short exposure image. In other examples, the HDR processing unit 106 may receive more than two images, for example, may receive a group of many different images, where different images in the group have different exposure levels.

[0034] In step S304, the image registration logic 202 performs image registration to align the images relative to each other. In this way, the image registration logic 202 geometrically and radiometrically aligns the images relative to each other to facilitate subsequent HDR processing. The first of the images is taken as a reference image and the other images are then geometrically and radiometrically aligned to the reference image. In some examples, the long exposure image (I L ) is taken as the reference image, while in other examples the short exposure image (I S ) is taken as a reference image, and methods for geometrically and radiometrically aligning the images are known in the art and are therefore not described in detail herein.

[0035] Image I L and image I S The images are aligned radiometrically so that the intensities from the two images are on the same linear scale. For example, it may be assumed that the image data is available from the image processing block 104 in a linear form and that the exposure time and sensitivity setting are known for the images. The images can then be aligned radiometrically by multiplying their intensity values ​​by a relative exposure factor. For example, given an exposure time t and a sensitivity or ISO setting i for a first image I1 (which is taken as a reference image) and a second image I2, the second image can be aligned radiometrically with the reference image according to the following equation:

[0036]

[0037] Geometric alignment attempts to deal with the motion that may occur between the moments of capturing different images (motion of objects in the scene or motion of the camera), but defects may occur in the alignment process, so that the images are not fully aligned. As an example, geometric alignment can be based on the assumption that the camera is kept approximately still between the moments of capturing the images. Camera motion can be attributed to the user's failure to keep the camera completely still and significantly includes rotations of up to about 5% of the field of view. A projective homography matrix model can be used to describe the transformation that can align the images. As an example, the projective homography matrix can be estimated using a local Lucas-Kanade (LK)-based image registration method in a coarse-to-fine pyramid configuration. Although it may be possible to estimate and compensate for local scene motion, intensive motion compensation is computationally expensive and motion compensated errors will still have to be concealed. Therefore, such motion compensation is not implemented in the method described in this article, which is suitable for use on devices with limited processing resources (e.g., mobile devices). As an example, in order to reduce the computational cost, the geometric alignment process can be applied to only the green channel of the image at half the resolution of the full image. Alignment may be applied at different scales, for example starting at a coarse scale and then proceeding to finer scales, whereby the results at the coarse scale are used to initialize the alignment process at the finer scales.

[0038] As mentioned above, imperfections may occur during the geometric alignment process, so that there may be inconsistencies in the aligned images. These inconsistencies can be masked as described below so that in the image regions where inconsistencies occur, the fusion process samples image values ​​from only one of the images when forming the fused image.

[0039] When the short exposure image (I S ) and long exposure images (I L ) are combined (ie, fused) to form an HDR image (IR), the fusion process preferably combines the images from the two images I S and I L The best available information is combined. Generally, long exposure images have a better signal-to-noise ratio (SNR) than short exposure images, but are more likely to be saturated. When forming an HDR image, there are a number of rules that are followed in the examples described in this article:

[0040] (i) For saturated image areas, the short exposure image is used (and the long exposure image is not used) to form the corresponding image area of ​​the HDR image, to thereby reduce the unwanted saturation effect.

[0041] (ii) To avoid ghosting effects, any connected region containing motion is extracted from only one image (I S or I L) sampled. In this way, all displays associated with the same moving object are sampled from the same image to avoid discontinuous motion artifacts. This is done by connecting adjacent motion regions to each other and to saturated regions. Motion regions connected to saturated regions are sampled from the short exposure image (rather than the long exposure image), while motion regions not connected to saturated regions are preferably sampled from the long exposure image (rather than the short exposure image). A more detailed description of what the term "connected" means in this context is provided below.

[0042] (iii) If it is compatible with the above two requirements, the image region of the HDR image is preferably formed using a long exposure image rather than a short exposure image, because a long exposure image generally has a higher SNR than a short exposure image.

[0043] (iv) The transitions between images used to form image regions in the HDR image are smoothed over a series of image regions. In this way, the transitions are gradual. This is because abrupt transitions between source images can result in noticeable seams in the HDR image and will reveal small registration defects and differences in image characteristics. Therefore, the images are smoothly blended to hide the transitions.

[0044] The aligned images are passed from the image registration logic 202 to the gradient determination logic 204 , the saturation determination logic 206 , the motion determination logic 208 , and the image fusion logic 228 .

[0045] In step S306, the saturation determination logic 206 determines which image regions in one or both of the images are saturated. The long exposure image has a higher exposure level than the short exposure image, so it is sufficient to consider the long exposure image (rather than the short exposure image) in order to determine which image regions are saturated. This is because it can be assumed that any saturated regions in the short exposure image will also be saturated regions in the long exposure image. Thus, as an example, saturated image regions are determined by determining for the long exposure image (I L ) is detected by checking whether the intensity value of each color channel of the image (I) is greater than a saturation threshold. Image regions with any channel above the saturation threshold are marked as saturated. In this way, a saturation mask is formed for each image region indicating whether the image region is saturated. The image region describes the image (I) at which S and I L ) can have image locations with image values. The image region can be an image with the same S and I L ) with the same resolution so that each image region corresponds to a corresponding pixel position. Alternatively, the image regions may be arranged at a resolution greater than that of the image (I S and I L) pixels, so that each image corresponds to a corresponding group of pixel positions. In this way, the filtering operation can be performed at a coarser resolution than the image (I S and I L ) pixels, a coarser resolution is applied to the image area.

[0046] In step S308, the motion determination logic 208 determines that between the two images I S With image I L The image region where there is motion between the two images is detected. Motion at the image region is detected by calculating an error metric between the image values ​​of the two images at the location of the image region. The "error metric" is the difference between the images. Note that although the two images initially have different exposure levels, the radiometric alignment performed in step S304 removes this difference so that motion detection can be performed between the two images without further radiometric registration.

[0047] The noise mean is removed from the image. A low pass filter will be used later to prevent noise from being detected as motion. To make this effective, the noise mean is preferably zero so that the local average (filtered) noise is zero. Image noise typically does not have a mean of zero, or the dark spots in the metadata may be incorrect. The sensor noise mean can be measured by capturing an image without exposing the sensor to light. In some examples, the sensor array 102 may provide an unexposed sensor area for measuring dark levels and noise for each captured frame. The resulting dark image I B The mean of the image is roughly the same as the image dark level b and the noise mean n μ Multiply by the sensor sensitivity (ISO setting) i related, such as mean (I B )=n μ i+b. By capturing dark images at different ISO settings, n μ and b can be recovered using a simple linear fit.

[0048] In a preferred example, the color channels are weighted according to their visual importance and to compensate for the different sensitivities of the image sensors of the sensor array 102 for each channel. The weighting matrix w may be, for example, based on the ITU-R BT.2020 weights for calculating luminance from RGB and the channel gains {g R , g G , g B} (where color correction and white balance have been applied and the gain will be 1), so that:

[0049]

[0050] Use the following equation to find the brightness normalized difference image D(x):

[0051] I′ L (x) = W(I L (x)-n μ ), (3)

[0052] I′ S (x) = kW (I S (x)-n μ ), (4)

[0053]

[0054] Among them, I′ R (x) is the noise-adjusted reference image, and I′ A (x) is the noise-adjusted non-reference image. For example, if the long exposure image I L is the reference image, then I′ R (x) = I′ L (x) and I′ A (x) = I′ S (x), and vice versa if the short exposure image is the reference image.

[0055] A low pass filter is applied to the difference image D(x) to reduce the effect of image noise. A recursive resampling filter is used to determine a filtered version of the difference image D f =f(D,n). For example, the image can be obtained by The image is downsampled by summing, and then recursively upsampled by a factor of 2 using bilinear interpolation to the original resolution. The amount of filtering required depends on the noise characteristics of the image. The error image e sse (x) is calculated from the filtered difference image Df by, for example, summing over the squares of the RGB channels according to the following equation:

[0056]

[0057] By comparing the square error image and (e sse (x)) A threshold is applied to identify motion regions. In this way, a motion mask is formed for each image region indicating whether there is motion in the image region.

[0058] In step S310, the type mask determination logic 212 combines the saturation mask and the motion mask into a 3-level mask, which is referred to herein as the "type mask" m(x). In the example described herein, in the type mask, saturated image regions have a value of 2, motion image regions have a value of 1, and image regions that have neither saturation nor motion have a value of 0. Therefore, the type mask includes a type indicator for the image region to thereby classify each of the image regions into one of a plurality of types: for example, saturated, motion, or empty. Saturated image regions are often detected as motion in the motion mask because saturation can cause a difference between the two images that is higher than the motion threshold described above. If an image region has both saturation and motion according to the saturation mask and the motion mask, the type mask indicates the image region as a saturated image region (giving it a value of 2 in the type mask). That is, saturation takes precedence over motion in the type mask. Therefore, the type mask indicates the maximum of the two values ​​where motion and saturation overlap.

[0059] Figure 5 An example type mask 502 is shown including a saturated region 5041 (shown in cross-hatching). The saturated region 5041 is surrounded by a motion region 5042 (shown in diagonal hatching). Saturated regions are often surrounded by regions that were initially determined to be motion regions because saturation causes the difference between the two images. Figure 5 The white areas in represent image areas that are neither saturated areas nor areas that include motion. The type mask 502 also includes two more areas 506 and 508. The motion area 508 is close to the saturated area 5041 and can therefore be connected to the saturated area 5041 when the connectivity filter unit 214 described below is implemented, while the motion area 506 is not close to the saturated area 5041 and therefore may not be connected to the saturated area 5041 when the connectivity filter unit 214 is implemented.

[0060] In step S312, the gradient determination logic 204 determines, for example, L and I S The image gradient indication g for each image region of the image provides an indication of the level of detail in the image region of the image. The image region for which the image gradient indication is determined may be the location of a pixel or pixel block of the image. For example, the image gradient indication g for the image region may be found by using the following formula from a 2×2 pixel block The normalized gradient magnitude of the green channel of each image is calculated at half the resolution of

[0061]

[0062] For long exposure images I LAll of the image gradient indicators of the image region x are combined into a matrix G L Similarly, for the short exposure image I S All of the image gradient indicators of the image region x are combined into a matrix G s These matrices are passed to the filter cost determination logic 210 .

[0063] In step S314, the filtering cost determination logic 210 determines the filtering cost based on the short exposure image I S and long exposure image I L The determined image gradient indication determines a filtering cost for the image region. The "filtering cost" of an image region is a weight used in applying filtering to an image region. That is, the filtering cost for an image region is a measure of the effect that the image region has on the attribute being filtered. For example, an attribute may be "closeness" in terms of whether two image regions are connected. In this case, the filtering cost of the image region can be used by a connectivity filter to define the closeness of the two image regions based on the sum of the filtering costs along the path between the two image regions. Since the filtering cost of the image region depends on the image gradient at the image region, the closeness of the two image regions depends on the image gradient along the path. In another example, the attribute may be "smoothness", such that a filtering cost defining the smoothness of an image region can be used by a fusion filter to determine the rate at which a fusion mask changes across image regions.

[0064] As described in more detail below, the filter cost is used in the filtering operation to thereby depend on the image I S and image I L The image gradient is compared with image I S and image I L The relevant image data can be filtered by determining the S and image I L The minimum gradient indication of the image area is then multiplied by the multiplication factor (b gain ) and then shift (b min ) is added to the result of the multiplication to determine the filter cost B(x) for the image region. In this way, the matrix B of the filter cost B(x) is determined according to the following equation:

[0065] B=b min +b 9ain min(G L , G S ). (8)

[0066] b minis a parameter that sets the minimum cost so that B(x) is not zero even in completely uniform regions to thereby ensure a soft and stable blend even in completely uniform regions. min can be set to about 0.001, which has been shown experimentally to be an appropriate value. gain is a parameter that sets an overall gain factor to control the rate at which fusion occurs over an image region relative to the level of detail in the image region. gain is set to approximately 0.25 and this performs well in experiments. In the examples described herein, the filter cost is in the range from 0 to 1, i.e., 0≤B(x)≤1. The filter cost for image areas that fall outside of one of the source images (where the images do not completely overlap) is set to 1. As will become apparent from the description below, this prevents fusion or connectivity propagation around the outside of the image. The filter cost can be downsampled so that the filter cost is at a lower resolution than the image area. This can reduce the amount of processing performed when the filter cost is used to perform filtering operations using the connectivity filtering logic 216 and / or the fusion filtering logic 224. In addition, the filter cost determination logic 210 receives indications of saturated image areas and motion areas, and sets the filter cost for these areas to 1 to prevent fusion through nearby saturated areas or motion areas.

[0067] In order for motion masking to get convincing results, all objects must be treated in a consistent way. Instead of performing high-level object segmentation (which is a very complex computer vision problem), saturated pixels and motion pixels are classified into connected sets. Since the goal is to assign all moving objects to short exposure images (where part of the object is saturated) or to long exposure images (where no part of the object is saturated), the problem can be simplified to a term of filling the saturated state into connected motion regions.

[0068] The motion mask often contains many small holes and lines that can demarcate objects due to the noise filtering step and coincidental image similarities. In order to bridge these gaps in the type mask, the connectivity unit 214 may downsample the type mask by a factor of 2 to reduce the resolution of the type mask, for example, to approximately 512 pixels, for the purpose of determining connectivity in the connectivity unit 214. The reduced resolution type mask m c Each image region in (x) corresponds to a c×c image region block in the original type mask m(x),

[0069]

[0070] Find the values ​​of the reduced-resolution mask such that:

[0071]

[0072] In this way, if the c×c image area block in the original type mask m(x) contains an indication of a saturated image area, then the reduced-resolution type mask m c The corresponding image region in (x) is marked as saturated (ie given the value 2); otherwise, if the block contains at least one moving image region, the type mask m with reduced resolution is c The corresponding image region in (x) is marked as motion (ie, given a value of 1); otherwise, the reduced-resolution type mask m c The corresponding image region in (x) is marked as empty (ie, given a value of 0).

[0073] The filter cost B(x) is downsampled to the type mask m with the reduced resolution by summing over the corresponding region. c (x) The same resolution, so that the filtering cost B after reducing the resolution c (x) By In the following description of the operation of the connectivity unit 214, references to the filter cost B(x) may be understood to refer to the reduced resolution filter cost B in the preferred example in which the resolution of the filter cost is reduced. c (x).

[0074] The low resolution 3-level type mask is passed from the type mask determination logic 212 to the connectivity unit 214. In addition, the filtering cost is passed from the filtering cost determination logic 210 to the connectivity unit 214. The connectivity unit 214 applies two processes (in steps S316 and S318) to the low resolution type mask to fill in saturation states on connected motion areas. The first is a filtering operation applied by the connectivity filtering logic 216 that is capable of connecting image regions across smooth image regions, and the second is a joint search filling algorithm implemented by the joint search logic 218 that connects adjacent pixels based on their type indicators. In the following description of the operation of the connectivity unit 214, references to the type mask m(x) may be understood to refer to the reduced resolution type mask m in the preferred example in which the resolution of the type mask is reduced. c (x), as described above.

[0075] In step S316, the connectivity filter logic 216 applies a filtering operation to the type mask using the determined filtering cost for the image region. By using the filtering cost as described above, the connectivity filter logic 216 depends on the image I L and image I SThe type mask is filtered by the image gradient of . The connectivity filtering logic 216 may apply a recursive filtering operation that determines a corresponding cumulative cost for each of the plurality of image regions by finding the lowest cumulative cost (of the filtering cost) along the path from the image region to the image region of interest. For example, the image region of interest may be a saturated image region, such that the cumulative cost for the image region is the lowest cumulative cost of the filtering costs along the path from the image region to the saturated region. The type indicator for the image region may be selectively set to indicate a saturated type depending on a comparison of the determined lowest cumulative cost with a threshold. This may also depend on the type of the image region, for example, if the type indicator is initially set to a motion type, the type indicator may be set in this manner. For example, a motion region that is found to be close to a saturated region is reclassified as a saturated image region in the type mask. The image region is "close" to a saturated region when the cumulative cost determined for the image region is less than the threshold. The filtering operation performed by the connectivity filtering logic 216 is intended to connect discontinuous image regions in a smooth image region such as the sky, for which the distance at which the transition between using different images to form an HDR image will be noticeable.

[0076] Figure 6 An example of a type mask 602 before a connectivity filtering operation and a type mask 604 after a connectivity filtering operation for an image with significantly low image gradients (e.g., an image of a cloudy sky) is shown. Figure 6 In 602, white image regions have saturation type indicators, patterned image regions have motion type indicators, and black image regions have null type indicators. It can be seen in 602 that motion is detected between the two images, and the motion regions are scattered (i.e., not fully connected), so the joint search process cannot connect all the motion image regions in the images. However, because the image (I S and I L ) have low image gradients, the filtering cost B(x) for these image regions is low, enabling the connectivity filtering logic 216 to bridge the gaps between the moving image regions in the cloudy sky and all of these moving regions to be connected so that they are all changed to the saturated type in the filtered type mask 604 because they can be connected to at least one saturated region. This is useful because the transitions between images used to form the HDR image will most likely be noticeable in some of the image regions (e.g., the region showing the cloudy sky) due to low image gradients in those image regions.

[0077] Figure 4An example of steps performed as part of step S316 to filter the type mask mm(x) is shown. The connectivity filtering logic 216 receives a type mask having a type indicator for indicating the type of the image region according to:

[0078]

[0079] The cumulative cost for the image region is determined using the filtering cost B(x). The cumulative cost can be referred to as a "distance" d(x), but it should be noted that these "distances" are weighted by the filtering cost and do not necessarily indicate the actual physical distance between the image region and the saturated region. Conceptually, a distance map for the distance d(x) is recursively calculated to map the distance to the nearest saturated image region. In step S402, if the first image region is saturated, that is, if m(0) = 2, then the cumulative cost (or "distance") d(0) for the first image region is set to a value of 0, otherwise, that is, if m(0) ≠ 2, then the cumulative cost d((0) for the first image region is set to a value of 1. The remainder of the cumulative cost is calculated as follows Figure 4 The connectivity filtering logic 216 shown in FIG.

[0080] Recursively implement for each image region Figure 4 4 to S418 shown in FIG. 4. That is, the filtering operation applied by the connectivity filtering logic 216 is a recursive filtering operation. As described in more detail below, in the examples described herein, recursive filtering operations are performed in a forward direction and in a reverse direction. The order in which image regions are processed may be different in different examples and may, for example, advance across the image in a diagonal manner. In step S404, the lowest cumulative cost d(x) for image region x is determined by adding the filtering cost B(x) of region x to the lowest cumulative cost of previously processed image regions. For example, the previously processed image region may be any one of the image regions having a lower cumulative cost just above or just to the left of image region x. This can be represented as:

[0081]

[0082] In step S406, it is determined whether the image region x has a motion type. That is, it is determined whether the type indicator satisfies m(x)=1. If so, the method passes to step S408, in which it is determined whether the cumulative cost d(x) for the image region x is less than a threshold T. In the example, the threshold T is set at , but in other examples, the threshold may be set at other values, such as . If it is determined in step S408 that d(x)<T, the method passes to step S410. Those skilled in the art will appreciate that in other examples, the test in step S408 may be whether d(x) is less than or equal to T rather than whether d(x) is less than T. In step S410, the type indicator for image region x is set to indicate a saturation type. That is, in the example described in detail herein, m(x) is set to a value of 2. The method proceeds from step S410 to step S412. In addition, if it is determined in step S406 that image region x does not have a motion type, the method passes from step S406 to step S412 without executing step S408 or S410. In addition, if it is determined in step S408 that the cumulative cost for image region x is not less than a threshold, the method passes from step S408 to step S412 without executing step S410.

[0083] Thus, steps S406 to S410 mean that if an image region that was originally a motion region (where m(x)=1) is "close" to a saturated region, the type indicator for the image region is set to indicate a saturated type (i.e., m(x) is set to have a value of 2), thereby connecting the motion region to a nearby saturated region. The "closeness" is determined by determining whether the cumulative cost for the image region is less than a threshold, so that the closeness takes into account the image gradient in the image. In this way, the motion region can be connected to the saturated region at a large physical distance if the image gradient is low at that distance. This can be beneficial because the transition between two regions of the image used to form the HDR image can be noticeable due to the low image gradient in the transition region. Conversely, the motion region may not be connected to the saturated region at a small physical distance if the image gradient is high at that distance. This is beneficial because the transition between two regions of the image used to form the HDR image can be noticeable due to the high image gradient in the transition region.

[0084] In step S412, it is determined whether the image region x has a saturation type, that is, whether m(x)=2 holds. If so, the method passes to step S414, in which the cumulative cost d(x) for the image region x is set to zero. The method passes from step S414 to step S416. In addition, if it is determined in step S412 that the image region x does not have a saturation type, that is, m(x)≠2, the method passes from step S412 to step S416 without executing step S414.

[0085] In step S416, it is determined whether there are more image regions to consider, and if so, the method passes to step S418, in which x is incremented by one. As described above, the order in which the image regions are processed may be different in different examples and the order in which the image regions are processed is determined in the manner in which x is incremented by one. For example, the image regions may be processed in a diagonal manner, starting from a corner of the image and proceeding in a diagonal line that is incrementally further away from the starting corner. The method then passes back to step S404 to repeat steps S404 to S418 for the next image region. If it is determined in step S416 that there are no more image regions to consider, the method passes to step S420, in which it is determined whether the method should be repeated for the image, and if so, the method passes back to step S404. For example, the method may first be performed in a forward direction (e.g., starting in the upper left image region and proceeding to the lower right image region), and then the method may be repeated in a reverse direction (e.g., starting in the lower right image region and proceeding to the upper left image region).

[0086] Some of the steps may be slightly different when the method is performed in reverse. For example, in the reverse direction, the method first considers the final image region (d(N), where N is the number of image regions) and then proceeds backwards toward the first image region to determine the cumulative cost. This second pass in decreasing order of x makes the process symmetric. When processing in reverse order, equation 11 used in step S404 is replaced with equation 12:

[0087]

[0088] Additionally, in step S418, x is decremented by one rather than incremented by one when processing in reverse order. If the connectivity filter logic 216 is applied multiple times, it may converge to a stable result, however two passes (e.g., one in the forward direction and one in the reverse direction) tend to be sufficient for processing smooth image regions as intended by the connectivity filter logic 216.

[0089] In other examples, rather than processing the image region in a diagonal manner, the method can be broken down into processing the x-direction and the y-direction, respectively, by using only one of the neighborhoods in equations 11 and 12 at a time. In these examples, twice as many passes over the image are performed, but these methods may be beneficial for certain implementations. For example, a GPU may more efficiently process many rows or columns in parallel, which (due to sequential dependencies) prevents information from being transferred in both directions simultaneously, but may be many times faster.

[0090] When (as determined in step S420) the method is no longer to be repeated, for example, after two passes, step S316 is complete and the method continues to step S318.

[0091] In step S318, the joint search logic 218 implements the joint search process to selectively set the type indicator to indicate the saturation type for image areas having the motion type and connected to image areas having the saturation type. In this way, the joint search filling process is used to take into account the connectivity in the carefully designed structure. Those skilled in the art will understand how to implement the joint search process. For example, the joint search process can be used to find a set of image areas connected by 8-neighborhood adjacency. If the connected area contains any saturated image areas, all image areas in the connected area are marked as saturated.

[0092] Figure 7 An example is shown including an image 702 including saturated regions near a carefully designed high contrast edge with motion in some regions. An initially downsampled type mask 704 is shown (e.g., as output by the type mask determination logic 212 in step S310), indicating saturated regions (e.g., white regions), motion regions (e.g., hatched regions), and empty regions (e.g., black regions). The type mask after the connectivity filter logic 216 has been applied is shown as 706, and it can be seen that a large number of motion regions have been converted to saturated regions, but some of the motion regions have not yet been connected to saturated regions in areas of high image detail (e.g., between branches in image 702). The type mask after the union-find logic 218 has been applied is shown as 708, and it can be seen that the connected motion regions from mask 706 have been set to the saturated type, with only the remaining two unconnected motion regions being motion regions. The union-find method is able to connect image regions in areas of high image detail (e.g., between branches) where the connectivity filter logic 216 was unable to connect due to high image gradients.

[0093] After step S318, the modified reduced-resolution type mask m is used. c (x) to update the original resolution type mask m(x). To do this, wherever the reduced resolution type mask m c The mask value in (x) has changed from 1 to 2, and this change is propagated back to the original type mask m according to the following equation c (x):

[0094]

[0095] The type mask is passed from the connectivity unit 214 to the fusion unit 220. For clarity, Figure 8(a) shows a graph representing the value of the type indicator 802 for only one row of the image region for the type mask m(x). In step S320, the initial blending mask determination logic 222 determines two initial blending masks n based on the type indicators from the type mask m(x). L (x) and n s Initial fusion factor of (x). Initial fusion mask n L (x) is a binary mask indicating whether the corresponding image region has a saturation type indicator in the type mask m(x). The initial fusion mask n s (x) A binary mask indicating whether the corresponding image region has the motion type indicator in the type mask m(x).

[0096] Figure 8 (b) The initial binary fusion mask n L The graph of (x) is shown as solid line 804b. The initial fusion mask n L (x) has the value 0 for image regions with saturation type in the type mask m(x) (i.e., n L (x) = 0 if m(x) = 2), and has a value of 1 for image regions that do not have a saturated type in the type mask m(x) (ie, n L (x) = 1 if m(x) ≠ 2). Similarly, Figure 8 (c) The initial binary fusion mask n s (x) is shown as a solid line 806 b The initial fusion mask n s (x) has a value of 0 for image regions with motion type in the type mask m(x) (i.e., n s (x) = 0 if m(x) = 1), and has a value of 1 for image regions that do not have the motion type in the type mask m(x) (ie, n s (x)=1if m(x)≠1).

[0097] Using a binary mask to merge images will introduce sharp edges in the final image. These are particularly intrusive in smooth moving scene areas such as faces and clouds. Specifically, the fusion unit 220 preferably only fuses more than one image to form pixels of the HDR image in image areas that are unsaturated and have no motion. Only short exposure images are used in saturated image areas, and only one of the input images, which is preferably but not necessarily a long exposure image, is used in connected motion areas. Two initial fusion masks are used to form an image fusion mask, which is used to combine the long exposure image with the short exposure image to form an HDR image. The image fusion mask is preferably roughly as smooth as the image. That is, the image fusion mask should have a transition as smooth as the image. For example, in the case where the image contains a sharp edge, the edge can be exploited to hide the transition in the image fusion mask. In addition, in the case where the saturated area and the unsaturated area are close to each other, the transition from one to the other is preferably smooth and monotonous.

[0098] Mask n S (x) for the saturated region and mask n L (x) for unsaturated areas. Each initial fusion mask is filtered individually and then combined to form an image fusion mask. In image areas where the type mask is non-zero, i.e., in saturated areas or motion areas, the filtering cost (i.e., fusion rate) is set to 1. In this way, the filtering cost is modified to B'(x) according to the following equation:

[0099]

[0100] Setting the fusion rate to 1 in image regions where the type mask is non-zero prevents the fusion process in one mask from extending beyond neighboring regions in other masks.

[0101] The filtering cost B′(x) of the image region x provides an indication of the smoothness of the image region. In step S322, the fusion filter logic 224 uses the filtering cost B′(x) to filter the initial binary fusion mask (n S (x) and n L (x)). In this manner, the corresponding transition is smoothed at a smoothing rate according to the filter cost for the image region at the transition in the initial blend mask. For example, the fusion filter logic 224 can be configured in a manner similar to the connectivity filter logic 216 to implement a filtering operation such as a recursive filtering operation using the filter cost, and the filtering can be performed multiple times, such as in a forward direction and then in a reverse direction. For example, the forward filtering operation can be performed by the fusion filter logic 224 at the initial blend mask (n according to the following equation L (x) and n S(x), generally referred to as n(x)), is performed on each of the image regions (starting with the first image region, x=0):

[0102]

[0103] A backward filtering operation can then be performed by the fusion filtering logic 224 on each of the initial fusion masks n(x) (starting with the final image region, x=N) according to the following equation:

[0104]

[0105] In this way, Figure 8 (b) using line 804 b The initial fusion mask n is shown L (x) is smoothed to form a solid line 804 sm The smoothed fusion mask is shown. Similarly, Figure 8 (c) using line 806 b The initial fusion mask n is shown S (x) is smoothed to form a solid line 806 sm The smoothed fusion mask is shown. sm and 806 sm The smoothed fusion mask shown is stepped to a value of 1 at positions x3 and x2, respectively. This is due to the modification of the filter cost so that B′(x2)=B′(x3)=1. The forward filtering operation processes the image in increasing order of coordinates and the backward filtering operation processes the image in decreasing order of coordinates. This can be a raster scan order, but more generally this can be any order that ensures that one or more adjacent input pixels have been processed before the current pixel. Multiple filters can be implemented in the fusion filtering logic 224 and the filters can be separated so that each filter processes only one direction, such as horizontally forward and backward or vertically forward or backward. The order of passing has a negligible effect on the structure. Using separate filters may be beneficial for certain embodiments. For example, a GPU can efficiently process many samples in parallel using separate filters.

[0106] In a preferred example, two forward filtering passes and two backward filtering passes are applied for each of the initial fused masks to ensure that the fusion is propagated around corners. Using two repetitions may not be enough for the mask to fully converge to a stable result, but the difference between the smoothed fused mask and the fully converged fused mask becomes visually inconspicuous after two repetitions, so this is not enough.

[0107] In step S324, the image fusion mask determination logic 226 determines the image fusion mask n based on the smoothed fusion mask n. L (x) and nS (x) is used to determine the image fusion factor α(x) of the image fusion mask. In the example described in this article, the image fusion factor is in the range from 0 to 1, that is, 0≤α(x)≤1. The image fusion factor α(x) for inclusion in the image fusion mask is determined based on the smoothed fusion mask so that the image fusion factor α(x) is zero or one for image regions with a saturation type or a motion type. This means that the input image (I L and I S ). This can be achieved Figure 8 As can be seen in (d), Figure 8 (d) shows the image fusion factor α(x) with line 808. It can be seen that in saturated image regions (e.g., between x1 and x2 and between x7 and x8), the image fusion factor α(x) has a value of zero, and in motion regions (e.g., between x3 and x4), the image fusion factor α(x) has a value of one. In image regions that are neither saturated regions nor motion regions (e.g., between x4 and x6), the image fusion factor α(x) preferably has a value of one, but this is only possible without causing sharp transitions. The transition in the image fusion factor α(x) is smooth, i.e., the image fusion factor α(x) has a finite first-order derivative.

[0108] The image fusion factor α(x) can be determined according to the following equation:

[0109]

[0110] According to Equation 15, the image fusion factor α(x) follows the smoothed fusion factor n L (x), unless the motion region is close to the saturation region, so that the transition in the image fusion factor α(x) must be larger than that in the smoothed fusion factor n L (x) appears faster. For example, in the region between x2 and x3, the image fusion factor α(x) appears faster than the smoothed fusion factor n L (x) transitions from 0 to 1 at a faster rate (it must step to a value of 1 at x3 to avoid gradually intruding into the motion region). In the region between x2 and x3, Therefore, the image fusion factor α(x) is obtained The value of α(x) is set to ensure that the transition of α(x) is smooth and does not extend into the saturation region (between x1 and x2) or into the motion region (between x3 and x8). In other regions (e.g., between x0 and x2 and between x3 and x8) Therefore, the image fusion factor obtains the smoothed fusion factor n LFor example, the motion region (between x3 and x4) is not close to the saturation region (between x7 and x8), so the smoothed fusion mask n S (x) can be attributed to the fusion filter logic 224 in the smoothed fusion mask n L The point at which (x) begins to transition from 1 (at x6) to 0 (at x7) is the point at which (x) transitions from 0 (at x4) to 1 (at x7). 75 Therefore, in these image regions, the image fusion factor α(x) can be simply obtained by smoothing the fusion factor n L The value of (x).

[0111] The image fusion mask α(x) is passed to the image fusion logic 228, which also receives the input image I S and I L The image fusion mask is used to transform the input image (I S and I L ) are fused to form a fused image (IR).

[0112] Specifically, in step S326, the image fusion logic 228 upsamples the image fusion mask α(x) by interpolation by a factor of 2 so that it is consistent with the input image (I S and I L ) of the same resolution. Then, in step S328, the image fusion logic 228 uses the upsampled image fusion mask α(x) to determine the image fusion factor for the image (I S and I L ) is used in the weighted sum of L (x) and I S The weights of (x) are α(x) and (1-α(x)) respectively. Therefore, the fused image (IR) can be determined according to the following equation:

[0113] I R (x) = α(x)I L (x)+(1-α(x))I S (x). (16)

[0114] As described above, the input image I S and I LThere may be different exposure levels (which may be captured at different times) and the fused image IR may be an HDR image. In some examples, there may be more than two images to be fused to form an HDR image, wherein the images are iteratively combined in pairs in an order of exposure levels according to the pattern, so that the fused image ("reference image") resulting from fusing two of the images in a first iteration is fused with another of the images in a later iteration.

[0115] For example, if the images are processed in decreasing order of exposure values, the first two images can be combined to form a reference image assigned the exposure value of the shorter of the two exposures, and then the reference image is fused with another of the images, and so on, until a final fused HDR image is formed. If the images are processed in increasing order of exposure values, the reference image is assigned the exposure value of the longer of the two exposures used to form the reference image.

[0116] Thus, if more than two images are to be combined, the images are processed in a monotonic order of exposure levels (e.g., in a monotonic order of (exposure time) x (sensitivity)) so that it is easy to determine which of the reference image and the new image is the longer exposure and the shorter exposure. The first image is used as the initial reference image and defines the reference coordinate space. Each new image is in turn geometrically and radiometrically aligned with the reference image and fused with the reference image to produce an updated reference image. Inconsistencies between the new image and the reference image are masked by modifying the alpha mask so that inconsistent areas are sampled from only one image. A sequential processing pipeline may be selected to avoid the need to have all images in memory at once.

[0117] The fused HDR image (IR) is output from the HDR processing unit 106 and may be provided back to the image processing unit 104, which may perform further processing of the image to determine a processed image. The processed image may be used for any suitable purpose, such as being displayed on a display, sent to another device, or stored on the memory 108, etc.

[0118] In the examples described above, two or more images are combined to form a blended HDR image for use in high dynamic range image processing. More generally, the above method can be used to combine images for any appropriate purpose, where it can be beneficial to apply a filtering operation that uses a filtering cost that indicates an image gradient in the image. For example, multiple images with different focuses are combined according to the examples described herein. Multiple images can be captured continuously, where the focus setting is set differently for the capture of each image, but the exposure level can be the same for each of the images. Note that if the images are captured continuously (i.e., at different times), there may be motion between the different images, similar to as described above with respect to the capture of different images for use in forming an HDR image. In the example of multiple images with different focuses, different image regions may be in better focus in the different images, and the images can be combined to form a blended image, where the better focused image regions are preferentially selected from the images. The transition between image regions obtained from different images can be smoothed according to a filtering cost that depends on the image gradient in the image. For example, similar to as described above, the filtering cost can be used in filtering operations, such as (i) in a connectivity filtering operation for determining connected image regions obtained from the same image (note, as described above, for image regions where motion exists, a single image should be used to form a fused image to avoid motion artifacts), and / or (ii) in a fusion filtering operation for fusing two or more images together to form a fused image, so that the transition between the images used to form the fused image is smoothed.

[0119] In other examples, the filtering principles of the methods described above can be used for purposes other than combining images. For example, a filtering operation using a filtering cost that is indicative of image gradients in the image can be useful for filtering a single image. For example, a filtering operation can be used to apply denoising to an image, such that the extent of denoising applied to an image region depends on the filtering cost for the image region. This can be useful because how noticeable the filtering is can depend on the image gradients in the image. For example, denoising filtering can be more noticeable in highly detailed image regions and less noticeable in less detailed image regions, and thus the amount of denoising can be increased for image regions with high levels of detail compared to image regions with low levels of detail.

[0120] Fig. 9906. A computer system is shown in which the HDR processing unit 106 may be implemented. The computer system may include a GPU 902, a CPU 904, a memory 906, and other devices 908, such as a display 910, a speaker 912, and a microphone 914. The computer system may also include a camera sensor 916, which includes a sensor array 102 and a camera pipeline including an image processing unit 104. As described above, the HDR processing unit 106 may be implemented on the GPU 902. The components of the computer system may communicate with each other via a communication bus 920. The HDR processing unit 106 may be implemented (e.g., in hardware) as shown. Fig. 9 904. Alternatively, the HDR processing unit 106 may be implemented on the CPU 904. If the HDR processing unit 106 is implemented in software, it may be stored in the memory 906 as a computer program code and may be executed on a processing unit in a computer system (e.g., on the GPU 902 or the CPU 904).

[0121] Generally, any of the functions, methods, techniques, or components described above (e.g., HDR processing unit 106 and its components) can be implemented in modules using software, firmware, hardware (e.g., fixed logic circuitry), or any combination of these implementations. The terms "module," "function," "component," "block," "unit," and "logic" are used herein to generally represent software, firmware, hardware, or any combination thereof.

[0122] In the case of software implementation, a module, function, component, block, unit or logic represents a program code that performs a specified task when running on a processor (e.g., one or more CPUs). In one example, the method described can be performed by a computer configured with software stored in a machine-readable form on a computer-readable medium. One such configuration of a computer-readable medium is a signal-bearing medium and is therefore configured to send instructions (e.g., as a carrier wave) to a computing device, for example, via a network. The computer-readable medium can also be configured as a non-transient computer-readable storage medium and is therefore not a signal-bearing medium. The example of a computer-readable storage medium includes a random access memory (RAM), a read-only memory (ROM), an optical disk, a flash memory, a hard disk storage, and other storage devices that can be accessed by a machine and can use magnetic technology, optical technology, and other technologies to store instructions or other data.

[0123] The software may be in the form of a computer program comprising computer program code for configuring a computer to perform the constituent parts of the described method or in the form of a computer program comprising computer program code modules suitable for performing any of the entire steps of the method described herein when the program is run on a computer, and wherein the computer program may be implemented on a computer readable medium. The program code can be stored in one or more computer readable media. The features of the technology described herein are platform independent, meaning that the technology can be implemented on various computing platforms with various processors.

[0124] Those skilled in the art will also appreciate that all or part of the functions, techniques or methods may be implemented by dedicated circuits, application specific integrated circuits, programmable logic arrays, field programmable gate arrays, and the like. For example, the modules, functions, components, blocks, units or logic (e.g., HDR processing unit 106 and its components) may include hardware in the form of circuits. Such circuits may include transistors and / or other hardware elements available in the manufacturing process. Such transistors and / or other elements may be used to form circuits or structures that implement and / or contain, by way of example, memories such as registers, flip-flops or latches, logical operators such as Boolean operations, mathematical operators such as adders, multipliers or shifters, and interconnects. Such elements may be provided as classic circuits or standard cell libraries, macros, or abstractions at other levels. Such elements may be interconnected in a specific arrangement. The modules, functions, components, blocks, units or logic (e.g., HDR processing unit 106 and its components) may include circuits that are fixed functions and circuits that can be programmed to perform one or more functions; such programming may be provided based on firmware or software updates or control mechanisms. In an example, the hardware logic has circuits that implement fixed function operations, state machines or processes.

[0125] It is also intended to include software that "describes" or defines the configuration of the hardware that implements the modules, functions, components, blocks, units or logic described above, such as HDL (hardware description language) software, such as used to design integrated circuits or for configuring programmable chips to implement desired functions. That is, a computer-readable storage medium can be provided, which has a computer-readable program code encoded thereon, and the computer-readable program code is used to generate an image processing system configured to perform any of the methods described herein or to generate an image processing system including any device described herein. That is, a computer system can be configured to generate a representation of a digital circuit based on the definition of circuit elements and data definition rules for combining those circuit elements, wherein the non-transitory computer-readable storage medium can have processor-executable instructions stored thereon, which, when executed at such a computer system, cause the computer system to generate an image processing system as described herein.

[0126] The terms 'processor' and 'computer' are used herein to refer to any device or portion thereof having processing capabilities enabling it to execute instructions or special purpose circuitry capable of performing all or part of a described function or method, or any combination thereof.

[0127] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. It will be understood that the benefits and advantages described above may relate to one example or may relate to several examples.

[0128] Those skilled in the art will readily appreciate that any scope or value provided herein may be expanded or changed without losing the effect sought. The method steps described herein may be performed in any appropriate order or simultaneously under suitable circumstances. Without losing the effect sought, any aspect in the example described above may be combined with any aspect in other examples of the description to form another example.

Claims

1. An image processing method, comprising: determining a filtering cost for an image region based on an image gradient indication for at least one image; and De-noising is applied to the at least one image using the determined filter cost such that an extent of the de-noising applied to an image region depends on the filter cost for the image region.

2. The image processing method according to claim 1, wherein applying denoising comprises: A recursive filtering operation is performed to determine a respective cumulative cost for each of a plurality of the image regions by finding a minimum cumulative cost along a path from the image region to an image region of interest. 3 . The image processing method according to claim 2 , wherein the recursive filtering operation is performed in a forward direction and in a backward direction.

4. The image processing method according to any one of claims 1 to 3, further comprising: determining a type mask comprising type indicators for image regions to thereby classify each of the image regions into one of a plurality of types; The applying of denoising comprises: for each of the plurality of image regions: determining a lowest cumulative cost along a path from the image region to an image region of a particular type according to the type indicator; as well as The type indicator for the image region is selectively set to indicate the particular type depending on a comparison of the determined lowest cumulative cost to a threshold value. 5 . The image processing method according to claim 4 , wherein the selectively setting the type indicator for the image region to indicate the specific type further depends on the type indicator of the image region. 6 . The image processing method according to claim 4 , wherein there are three types: (i) saturation type, (ii) motion type, and (iii) empty type.

7. The image processing method according to claim 4, wherein applying denoising further comprises: A union-find process is implemented to selectively set the type indicator to indicate the particular type for an image region that has another type and is connected to an image region that has the particular type.

8. The image processing method according to any one of claims 1 to 3, further comprising: An image gradient indication is determined for the at least one image, wherein the filtering cost for the image region is determined based on the determined image gradient indication for the at least one image.

9. The image processing method according to any one of claims 1 to 3, wherein the at least one image is a single image.

10. An image processing system, comprising: a filtering cost determination module configured to determine a filtering cost for an image region based on an image gradient indication for at least one image; as well as A processing module is configured to apply denoising to the at least one image using the determined filtering cost, such that an extent of the denoising applied to an image region depends on the filtering cost for the image region.

11. The image processing system of claim 10, wherein applying denoising comprises: A recursive filtering operation is performed to determine a respective cumulative cost for each of a plurality of the image regions by finding a minimum cumulative cost along a path from the image region to an image region of interest.

12. The image processing system of claim 11, wherein the recursive filtering operation is performed in a forward direction and in a backward direction.

13. The image processing system according to any one of claims 10 to 12, further comprising: A gradient determination module is configured to determine an image gradient indication for the at least one image, wherein the filtering cost determination module is configured to determine the filtering cost for the image region based on the determined image gradient indication for the at least one image.

14. The image processing system according to any one of claims 10 to 12, wherein the at least one image is a single image.

15. A computer-readable storage medium having computer-readable code encoded thereon, the computer-readable code being adapted to perform the steps of the method according to any one of claims 1 to 3.

Images