Method and device for merging low dynamic range images into a single image

Abstract

The present invention discloses a method for constructing a high dynamic range image by combining low dynamic range images of different exposures. The method handles motion so as to minimize the chance of such "irrecoverable" occlusions and avoids the necessity of knowing or finding the response function of the low dynamic range image. In addition to this, the method is fast enough to be used in mobile camera devices and produces a good user experience.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more particularly to a software library for combining several images of the same scene taken at different brightnesses to achieve a single image representing a higher dynamic range of scene brightness. Background Art

[0002] Modern image sensors have insufficient dynamic range. They are unable to capture dark and bright areas in typical outdoor scenes (e.g. shadows and sky) with a reasonable signal-to-noise ratio. Typically, the pixel values ​​in dark areas are too low, comparable to the noise level, and / or the brightest areas are clipped to the sensor saturation level. To overcome this obstacle, few images with different exposure times and / or sensor sensitivity (gain) can be captured. Dark images (short exposure time) can capture bright areas such as the sky well, but will not contain useful information in dark areas. Bright images can capture dark areas (shadows) well, but bright areas will be overexposed and saturated. An algorithm is then needed to combine (stitch, fuse) the dark and bright images, extracting useful information from each image and generating a single high dynamic range image (HDR).

[0003] Many articles describe HDR fusion algorithms. Fusion can be achieved in different ways, which can be divided into two main approaches:

[0004] (1) Fusion in the radiation domain.

[0005] The fused image pixel values ​​are proportional to the scene brightness. These methods are practically applicable only to unprocessed images from the image sensor. Modern cameras perform complex processing on the sensor image, making it difficult or impossible to recover the overall response function of the camera afterwards.

[0006] This approach requires post-processing through complex adaptive local and global tone mapping algorithms to reduce the pixel value range in the fused HDR image to a displayable range.

[0007] (2) Fusion in the image domain.

[0008] A milestone in this approach was the article "Exposure Fusion" by Mertens et al. Different regions from differently exposed images with different camera response curves are stitched directly. To avoid brightness transitions between regions taken from different images, a Laplacian pyramid is used for fusion. This approach combines implicit local and global tone mapping, so that the fused image has a similar pixel dynamic range as the input images, so no complex post-processing is needed to reduce it to displayable levels.

[0009] Regardless of the method that will be used for fusion, the main problem in HDR fusion is moving cameras and moving objects when capturing multiple images. If the fusion algorithm does not take this into account, ghost-like artifacts will appear in the final image. Many methods exist to detect potential ghosting areas. Most of them rely on the assumption that the static parts of the scene are similar in the radiometric domain between the differently exposed images. The result of ghost detection can be a strong mask of similarity and dissimilar areas, or a weight per pixel per image showing the probability that each pixel contains a static scene.

[0010] By aligning the input frames using a frame registration algorithm, non-similarity due to camera motion is mostly eliminated. However, many other dissimilarities still exist in real scenes - they are caused by moving objects, motion blur, imperfect frame alignment, differences caused by different image processing parameters applied to differently exposed input images, and translational motion of the camera, resulting in mutual object movement within the image that cannot be represented and compensated by global frame alignment. Once detected, two main methods are used to avoid ghosting:

[0011] (1) Extracting motion regions from a single image

[0012] This method relies on the fact that the scene is self-consistent in each single image. The simplest approach is to extract all non-similarities from a reference image. If the exposure is bad in the reference image, this approach will make many moving areas overexposed or too dark or noisy. A better approach is to extract each non-similar region from an image or a set of images that are similar within that region.

[0013] (2) Global optimization to adjust the contribution weight of each pixel

[0014] The fusion of some images can be represented as a weighted average of their pixels, a fused pixel value, or a transformation of the images (such as a Laplacian pyramid or a gradient map). Each pixel in each image has its own weight. Ideally, in static areas, pixels with the best exposure should have the largest weight. In moving areas (non-similarity), only pixels from images that contain the background and are well exposed should have high weights. The most sophisticated ghost detection and removal methods try to define a global objective function that combines ghost detection information and depends on the weights. The weights are determined by minimizing this objective function. This is a very computationally intensive process and is not suitable for mobile devices, where computing power is limited and processing time is also limited by user expectations.

[0015] Hasinoff et al. take a completely different approach for HDR fusion and ghosting handling in “Burst Photography for High Dynamic Range and Low-Light Imaging on Mobile Cameras”. Instead of capturing images at different exposures, they capture multiple dark images to minimize overexposed areas, and apply complex multi-frame noise reduction to reduce the noise in the dark areas, thereby increasing the dynamic range. After denoising, the dark areas are brightened by multiplying the pixel values, which again increases the noise in the dark areas. Moving objects are handled by constructing a motion field. A large number of dark images are needed to get a good signal-to-noise ratio in the dark areas - they limit it to 8, probably for performance reasons. While they achieve very good overall results, the dark areas look noisy, and their approach is limited to the Bayer image domain, where the information in the darkest parts of the scene is not affected by image processing and quantization. This makes the algorithm unsuitable for post-processing steps.

[0016] In general, all methods that attempt to produce a fused image by a per-pixel weighted average of several images assume that there are many input images and that each pixel is well exposed in more than one input image. In practice, as few images with different exposures as possible should be captured to minimize processing time. Furthermore, the initial image alignment is never perfect. Even sub-pixel misalignment of the edges of objects in the images can lead to a noticeable degradation in sharpness after averaging. Alignment itself degrades sharpness somewhat due to the use of bilinear pixel interpolation, which is the only option available in contemporary mobile GPUs due to their frame warp speed. In order to preserve the sharpness of the input images as much as possible and minimize the number of input images required, each pixel in the fused image should be taken from one input image.

[0017] Another problem related to ghosting is that of unrecoverable regions. These are areas in the scene, usually occlusions caused by moving objects, that are badly exposed in all images. For example, if a dark object moves in front of a bright background, it must be taken from some of the bright images, in the place where it was. The bright background should be taken from the dark image, however, part of it is occluded by the dark object and is in a different position in the dark image. There has been little or no discussion of this issue in the literature so far. Summary of the invention

[0018] In order to solve the above technical problems in the prior art, in a first aspect, the present invention provides a method for merging low dynamic range images into a single image, the method comprising:

[0019] Acquire low dynamic range images with different EV compensation values;

[0020] Detecting non-similar regions of the image by equalizing the image brightness according to their mutual two-dimensional histograms;

[0021] modifying the image in non-similar regions before performing Laplace decomposition on the image;

[0022] Different fusion weights are reallocated and adjusted on the images to fuse a single high dynamic range image.

[0023] According to one embodiment of the present invention, image exposure and sensitivity settings are taken in a specific order: dark-normal-bright-normal-dark-normal-bright or bright-normal-dark-normal-bright-normal-dark.

[0024] According to one embodiment of the invention, the images are aligned to one of them by equalizing the different exposure brightnesses based on the image histogram and the mutual gamma calculated based on the tracking features within the equalized image.

[0025] According to an embodiment of the present invention, after brightness equalization is performed on the aligned images using mutual gamma values ​​derived from mutual two-dimensional histograms, non-similar regions in the images are found.

[0026] According to one embodiment of the present invention, each non-similar region is taken from its reference image, converted to the other two exposures using mutual gamma, and copied to part or all of the image at the corresponding exposure.

[0027] According to an embodiment of the present invention, the fusion weights in the non-similar regions are redistributed between the new local reference image of each region and the regions in the adjacent images with different exposures that are similar to the region.

[0028] According to one embodiment of the present invention, weights are adjusted after decomposition in a Gaussian pyramid to adjust global tone mapping and local contrast enhancement.

[0029] According to one embodiment of the present invention, the method further comprises: performing frame alignment on an image formed by capturing frames during the burst capture.

[0030] According to one embodiment of the present invention, performing frame alignment also includes: equalizing the brightness of the images through corresponding mutual gamma correction; detecting features to be tracked in the reference image and tracking the features in each other image; performing a homography transformation to warp each image from the other images to match the reference image.

[0031] According to one embodiment of the present invention, the method further comprises:

[0032] Seven input low-resolution images were captured at three exposure and sensitivity settings;

[0033] The third and fifth images are converted to the pixel brightness values ​​of the fourth reference image by mutual gamma;

[0034] The two difference masks are combined by an OR logic operation, and adjacent non-similar pixels are grouped together to form different regions using connected component labeling;

[0035] The pixel values ​​of each region are analyzed to determine which exposure should be used to acquire each region.

[0036] According to one embodiment of the present invention, the method further comprises:

[0037] Before pyramid decomposition, non-similar regions are replaced with brightness adjustment information from other images.

[0038] In another aspect, the present invention provides an apparatus for merging low dynamic range images into a single image, comprising:

[0039] An acquisition module, used for acquiring low dynamic range images with different EV compensation values;

[0040] a detection module for detecting non-similar regions of the image by equalizing the image brightness according to a mutual two-dimensional histogram;

[0041] a modification module for modifying the image in the non-similar region before performing Laplace decomposition on the image;

[0042] The fusion module is used to reallocate and adjust different fusion weights of images to fuse a single high dynamic range image.

[0043] In another aspect, the present invention provides an electronic device, comprising:

[0044] Processor; and

[0045] A memory for storing processor-executable instructions, wherein the processor is configured to execute the method described in any implementation of the first aspect.

[0046] On the other hand, the present invention provides a non-transitory computer-readable storage medium storing executable instructions, which, when executed by an electronic device with a touch-sensitive display, enables the electronic device to execute the method described in any implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 The data flow and main functional blocks of the entire HDR stitching algorithm are shown;

[0049] Figure 2 The sub-stages of the first processing stage are shown: the image ( Figure 1 #12 in the image) is aligned with one of them;

[0050] Figure 3 The sub-stages of “Difference and Decision” are shown;

[0051] Figure 4 The "Adapt to Image" sub-stage is shown;

[0052] Figure 5 is a block diagram of an apparatus for merging low dynamic range images into a single image according to an exemplary embodiment of the present disclosure;

[0053] Figure 6 Examples of apparatuses according to one or more embodiments of the present disclosure are provided. DETAILED DESCRIPTION

[0054] The present invention will now be described in more detail with reference to the accompanying drawings.

[0055] Reference Figure 1 , describing the general stages of high dynamic range (HDR) stitching according to the present invention.

[0056] The first stage is capturing images using the EV bracketing feature (stage #10), present in most modern mobile cameras. This stage is missing if the HDR stitching algorithm is run on a non-camera device (such as a computer) and the images with different exposures are captured and fed to it by an external device. If the stitching algorithm is run on a camera device, it asks the camera to capture 7 images with different EV compensation values ​​as fast as possible. The EV compensation value determines how bright (positive values) or dark (negative values) the image should be relative to the image captured by the camera without EV compensation. The embedded camera automatic exposure control algorithm determines the actual exposure time and sensor sensitivity. An EV value of 1 means a 2x increase in total exposure; -1 means a 2x decrease. The total exposure is the exposure time multiplied by the sensor sensitivity. In practice, at least 3 images need to be captured with EV compensation values ​​of -2, 0 and +2, which means a total exposure time 4x shorter and longer than the "auto" image with EV=0. The values ​​+2 and -2 are example values. If the camera has good image processing and good noise reduction, the output dynamic range can be further increased using the +3 / -3 values. + / -2 EV compensation is sufficient to meet the shooting needs of most entertainment scenes.

[0057] The image with EV0 is used as the reference image - it contains most of the well-exposed scene, and these areas will be carried over from it to the final HDR image with as little change as possible. Only the too dark and overexposed areas in the reference image will be taken from the brighter (EV+2) and darker (EV-2) images, respectively. This approach guarantees that the HDR image looks similar to the non-HDR capture (EV0), but with improvements in dark and bright areas. This is an important requirement in practice, as in some scenes a significant change in the average HDR image brightness relative to the non-HDR image may be perceived as an "improvement" by some users and a "deterioration" by others.

[0058] Using only 3 images leads to the frequent presence of unrecoverable regions (see the end of the Background section). To avoid this problem, we capture the input images in a certain order. The seven input low-resolution images have three exposure and sensitivity settings, where the middle 3 images (3rd, 4th, 5th) are from all 3 different settings, the 1st and 2nd images have the same settings as the 4th and 5th images, and the 6th and 7th images have the same settings as the 3rd and 4th images. An example of such a sequence given as EV compensation values ​​is: -2, 0, +2, 0, -2, 0, +2. Groups of images with the same settings are often referred to as "exposures" in the following, for example a short exposure refers to either of the two images taken with EV-2.

[0059] The 4th image (EV0) is the reference image. If the scene is purely static, only the 3rd and 5th images are used to improve the reference image. However, in most cases, some objects move within the scene, creating dissimilar and possibly unrecoverable areas. EV bracketing is necessary to capture images quickly, up to 30 frames per second with contemporary 8-12MPix image sensors. During capture, objects usually move linearly, or at least along some simple curves without turning. Having two dark images, bracketing the reference image guarantees that any unrecoverable (well exposed, but occluded) areas in either one are not occluded by the other.

[0060] Each of the 3rd-5th images may have a "reference" for some area (see below). Some pixels in this area may need to be taken from other exposures. The above sequence guarantees that each of the 3 images in the middle has two images of all the other exposures surrounding it, so the chance of unrecoverable areas is minimized.

[0061] The captured images are fed to the next stage 12 "aligned frames" which Figure 2 This is shown in more detail in . Figure 2, the first sub-stage is 210 "Inter-gamma". Since the three exposed images have different brightness, two inter-response functions are determined based on the histograms of the middle three images. These are lookup tables (LUTs) that are used to convert pixel values ​​of the shorter exposure to pixel values ​​of the longer exposure between adjacent exposures. Each point in the LUT has an index and a value, representing the pixel value in two dark and light image pairs. Two corresponding pixel values ​​(dark, light) correspond to the same interval in the cumulative histogram. The LUT values ​​of missing pixel values ​​(if any) are interpolated from the adjacent existing pixel values.

[0062] The next sub-stage is 212 “Detect and track features”. Good features to be tracked are detected in the reference image and tracked in each of the other images after equalizing their brightness by corresponding mutual gammas. The result is a set of corresponding pixel coordinates between each image and the reference, seeing the same object. Tracking uses a 3-starge resolution pyramid for faster performance. Then (214, “Fit homography”) a homography transform is found, describing how to warp each image to match the reference image. The loop is run for several iterations to detect and eliminate potential errors in the tracking process as well as features detected on moving objects - outliers. The homography is determined from the features on the background static scene and is presented as a grid for each image, except the reference image. The vertices of each grid contain the horizontal and vertical pixel coordinates in one of the images, corresponding to a regular grid of vertices covering the reference image, i.e. those are the back-mapped grids. The frames are then warped using the GPU for fastest performance and lowest power consumption - 216 “Warp images”. Frame alignment is performed when capturing frames during burst capture, minimizing the button preparation time for the entire HDR capture process.

[0063] Figure 1 The next stage in the process is 14 “Differences and Decisions”. Figure 3 See . Figure 3 , the first sub-stage again constructs "mutual gammas". This time, these are calculated on the aligned images, using 2D mutual histograms. The two axes (X,Y) of the mutual histogram represent the bins of pixel values ​​in each of the two images. Each point or cell in the mutual histogram is a 2D bin, representing a range (bin) of pixel values ​​in the two images. Each 2D bin contains a number - the number of pixels that are located at the same position in both images and have a pixel value within the corresponding 1D bin. The 2D histogram can be viewed as a surface - the histogram value is the height of each 2D bin above the XY plane. The surface contains a central ridge whose XY coordinates define the mutual gamma curve. The 2D bins close to the ridge correspond to pixels that see the same scene, but are affected by noise. The 2D bins outside the ridge correspond to pixels that see different scenes in the two images - moving objects.

[0064] Figure 3 The next step in , is to find non-similar regions – 312. First, the 3rd and 5th images (dark and light) are transformed to the pixel brightness values ​​of the reference image (4th – normal image) through cross-gamma. In each pair (3,4 and 5,4), the pixels at the same position in RGB space that differ by more than a threshold are marked as non-similar pixels in a binary image called “difference mask” – they come from moving objects.

[0065] The next stage is 316 “Grouping”. The two difference masks (3,4 and 5,4) are combined by an “OR” logical operation. Adjacent non-similar pixels are grouped together using connected component labeling to form distinct regions. Regions that are too small are discarded. Then in 318 “Select Source Exposure”, the pixel values ​​in each region are analyzed and a decision is made - which exposure should be used to acquire each region. This approach ensures image consistency (no ghosting) within the non-similar regions. Note that the non-similar regions include the locations of moving objects in the three intermediate images. Taking all pixels within the distinct regions from a single image ensures that no ghosting occurs.

[0066] Figure 1 The result of the overall block 14 “Differences and Decisions” in . is a set of masks of non-similar regions, and each mask should be taken at each exposure - i.e. the reference image (3, 4 or 5) for each region. Note that some pixels in a region will not be well exposed in its reference image either - these pixels will be taken from other exposures that look similar to the reference image for that region. Thus, a region is assigned to a certain (reference) exposure, but not all pixels will be taken from that exposure.

[0067] Figure 1The next stage in . are the 20 "weights". Ideally, if the scene is static, only the middle 3 images will be fused into a single HDR image. Fusion can be thought of as a weighted average of the pixels in the same position of the 3 images. Each pixel in each image is assigned a weight coefficient, forming 3 weight images. The weights are determined based on the pixel brightness in the reference image, they do not depend on the difference map at this stage. The brightness of each pixel is calculated as the weighted average of R, G, B and the maximum of the R, G, B values ​​of the pixel. The four weights are manually adjusted. The weights of the reference image are determined by a trapezoidal LUT table, defining the relationship of the weight to the pixel brightness. For medium pixel brightness, the reference weight is 1, which means that such pixels will only be taken from the reference image. Below a certain threshold, the reference weight is smoothly reduced to 0, and the remaining 1 is assigned to the bright image weight, which means that the darkest pixels will only take the brighter image, and there are some transition areas along the pixel brightness where the pixel will be averaged from the normal image and the brighter image. Similarly, the reference weight above a certain threshold is reduced, shifting weight to the darker image. So far, only the middle 3 images have weights other than 0. If you fuse the middle 3 images with their weights, as currently calculated, if some moving objects have brightness in the transition area of ​​the trapezoidal weight LUT, ghosting will appear in the fused image.

[0068] The next stage (22, “Redistribution”) transfers the weights within the non-similar regions of the reference image from each region to the neighboring image at different exposures. Suppose we decide to take the non-similar region from the dark image (the 5th image, EV-2) because most of it contains moving objects that are overexposed in the reference image. This region consists of the moving object and the background, which is occluded at the location of the object in the reference image:

[0069] The occluded region is part of the background and is too dark to be taken from the dark image. However, it is occluded in the reference image, so it should not be taken from there either. In this case, the occluded region should be taken from the 6th image, which has the same exposure as the reference image (EV0), but was captured after it. Since the object moves to the right, this region contains well-exposed background in the 6th image and can be taken from it. More generally, within each non-similar region, pixels with good exposure in its local reference image are assigned to that image (high weights for that image). Badly exposed pixels are assigned to neighboring images with other exposures that are similar to those in the local reference image. To calculate the similarity between pixels of different exposures, the mutual gamma from stage 14 is used. As a result, the weights in the non-similar regions that were distributed only in the middle 3 images after stage 20 are now redistributed to each region's new local reference image and to the regions similar to it, between its four neighboring images.

[0070] Reassigning weights is not enough to avoid ghosting. Fusion is done using a Laplacian pyramid. While decomposing each image into a pyramid, information from occluded areas (where weights were redistributed) is also propagated in well-assigned areas. As a result, part of the information that should be excluded in the fused HDR image actually appears there. To avoid this, in stage 16 “Adaptation to image” each non-similar area is taken from its reference image, transformed to two other exposures, using the mutual gamma from stage 14, and the exposure is replaced in all images accordingly.

[0071] The adapted image is then decomposed into a Laplacian pyramid in stage 18, “Decompose the image.” The corresponding weights are decomposed into a Gaussian pyramid in stage 24, “Decompose the weights.”

[0072] Stage 26 “Adaptation Weights” adjusts the ratio between local and global tone mapping. In general, there are two ways to compress the range of pixel values ​​of an HDR image to a displayable low dynamic range - local and global tone mapping (or local / global brightness and contrast enhancement). Global tone mapping roughly means transforming all pixels through a lookup table (gamma) - all pixels are the same. This method preserves the order of pixel brightness throughout the image, but reduces the contrast within bright areas. Local tone mapping means brightening the dark areas and darkening the bright areas while preserving the contrast therein, while reducing the difference between areas across the board. This method can produce good image contrast, but it often looks unnatural.

[0073] Fusion of images of different exposures using weights based on pixel brightness via a Laplacian pyramid has the effect of local tone mapping. If fixed weights are used (one number per image), fusion directly or via a Laplacian pyramid is equivalent to global tone mapping. The weight values ​​and the input image response curve determine the final response curve. To adjust the amount of local contrast enhancement over the global tone mapping, the weights at the coarsest level of the pyramid are pulled toward the same value, such as:

[0074] new_weight=weight*(1-k)+fixed_value*k

[0075] Manually adjust "fixed_value" and "k" to match the desired contrast. When "k" equals 1, all weights are equal and the fused image is globally tone mapped. When "k" decreases to 0, the fused image becomes increasingly detailed and contrasty.

[0076] The next stage 28 “fusion” averages the corresponding pixels in all image Laplacian pyramids according to the weights in the Gaussian pyramids. The result is a single Laplacian pyramid. During the weighted averaging, the intermediate layers of the resulting Laplacian pyramid are scaled by some manually adjusted multipliers, which has the effect of additional edge enhancement and additional local contrast enhancement. In addition, the finest layers of the final Laplacian pyramid, with smaller absolute values, are pulled further towards 0. This has the effect of denoising. The scale and thresholds of these operations can be adjusted depending on the desired appearance of the output image. The resulting Laplacian pyramids are then combined into a single image.

[0077] The final stage 30 “Post-Processing” corrects some issues that are inherent to using a Laplacian pyramid for fusion. Depending on the scene, after fusion and combining the pyramids onto the image, the pixel values ​​of the darkest areas may differ from those on the bright input image. The brightest areas taken from the dark image may also differ from their original values. Finally, the mid-tones taken from the reference image may differ from the original image. The black level of the fused image and the dark image (in the 3 intermediate images) is measured as a pixel value below which a certain percentage of all pixels are. The white level of the fused image is also measured as a pixel value above which a certain percentage of all pixels are. The average brightness of the fused image and the reference image is also measured. The measured values ​​determine a global tone mapping curve that is applied to the fused image and drives the dark levels to one of the bright exposures, pulls the white levels close to saturation, and increases the average brightness to the reference level if necessary, thus preventing the fused image from appearing darker than the reference image. These adjustments are intended to avoid obtaining a fused image whose brightness distribution differs too much from the reference image. In a typical camera setup, a user can capture the same scene (generating a reference image) both with HDR stitching and without HDR. Users expect the HDR image to be similar, but better than the reference image. Differences that are too large may be perceived as "worse" or at least "unexpected" behavior of the HDR stitching.

[0078] Typical embodiments of the present invention are described above as additional features for conventional mobile camera devices. Other embodiments may place different requirements on the HDR fusion algorithm and / or adjustments. They can be achieved by modifying certain processing stages. For example, if the requirement to look similar to the reference image is relaxed, the final post-processing (30) can be modified to construct the gamma in a more complex way. In another embodiment, it may be beneficial to use only two exposures - for example for performance reasons, to reduce the number of images captured and processed. In addition, if the dark areas are dark areas using an image sensor with large pixels and therefore a good signal-to-noise ratio, the algorithm in the present invention can enhance only the bright areas and overexposed areas, requiring only normal exposure and dark exposure. The same algorithm can be used, simply removing the part that processes the third exposure and adjusting the weight LUT and other adjustment parameters accordingly.

[0079] In another embodiment, further different from the typical embodiment, such as for machine vision consumers, an output HDR image with linear pixel response to light and a linear input image may be desired. Again, the same algorithm can be used, simplified by removing the mutual gamma (replacing them with simple multipliers) and removing the gamma curve in a post-processing stage (leaving only the black and white level adjustments).

[0080] See Figure. Figure 5 , a device 50 for combining low dynamic range images into a single image, comprising: an acquisition module 501, for acquiring low dynamic range images with different EV compensation values; a detection module 502, for detecting non-similar areas of the images by equalizing the image brightness according to a mutual two-dimensional histogram; a modification module 503, for modifying the images in the non-similar areas before performing Laplace decomposition on the images; and a fusion module 504, for reallocating and adjusting different fusion weights to the images to fuse a single high dynamic range image.

[0081] Figure 6 An example of an apparatus according to one or more embodiments of the present disclosure is provided. As shown in FIG. Figure 6 As shown, the device includes a central processing unit (CPU) 61, a storage medium 62, and a bus 63 connecting the CPU 61 and the storage medium 62. The storage medium 62 can store computer executable instructions. When the computer executable instructions are executed by the CPU 61, the device can perform the method disclosed in the present disclosure.

[0082] The present disclosure may include dedicated hardware implementations, such as application specific integrated circuits, programmable logic arrays, and other hardware devices. Hardware implementations may be constructed to implement one or more methods described herein. The applications of the apparatus and systems that may include various examples may widely include various electronic and computing systems. One or more examples described herein may use two or more specific interconnected hardware modules or devices to implement functions, wherein related control and data signals may communicate between modules and through modules, or as part of an application specific integrated circuit. Therefore, the disclosed computing system may include software, firmware, and hardware implementations. The term "module," "submodule," "unit," or "subunit" may include a memory (shared, dedicated, or group) storing a code or instruction that may be executed by one or more processors.

[0083] It should be understood by those skilled in the art that any one of the above embodiments is only an example, and does not mean that the present disclosure (including claims) is limited to these examples. According to the present disclosure, the technical features of the above embodiments or different embodiments can be combined, and the steps can be arranged in any order. There are many other variations of the above different aspects of the present disclosure, and for the sake of simplicity, these variations are not described in detail.

[0084] For simplicity of illustration and discussion, and for ease of understanding of the present disclosure, the accompanying drawings may or may not show known couplings of power / ground of integrated circuits (ICs) and other components. In addition, in order to avoid difficulty in understanding the present invention, the device can be represented by a block diagram, considering that the detailed features of the embodiments of the device shown in the block diagram depend largely on the implementation platform of the present disclosure (i.e., the detailed features can be fully understood by those skilled in the art). In the case of illustrating the detailed features (e.g., circuits) to describe exemplary embodiments of the present disclosure, it is obvious to those skilled in the art that the present disclosure can be implemented without the detailed features or in the absence of the detailed features, and the conditions under which the detailed features can be changed. Therefore, these descriptions should be understood to be illustrative rather than restrictive.

[0085] Although the present invention has been described in conjunction with the detailed embodiments of the present invention, various substitutions, modifications and variations of these embodiments will be apparent to those skilled in the art from the above description. For example, other storage structures (e.g., dynamic RAM (DRAM)) may employ the embodiments discussed.

[0086] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the scope of the appended claims. Therefore, within the spirit and principle of the present invention, any omission, modification, equivalent substitution, modification, etc. should be included in the protection scope of the present invention.

Claims

1. A method for merging low dynamic range images into a single image, the method comprising: Acquiring low dynamic range images with different EV compensation values, including: image exposure and sensitivity settings are shot in a specific order: dark-normal-bright-normal-dark-normal-bright or bright-normal-dark-normal-bright-normal-dark; Detecting non-similar regions of the image by equalizing the image brightness according to their mutual two-dimensional histograms; Modifying the image in the non-similar region prior to performing Laplace decomposition on the image includes: replacing the non-similar region with brightness adjustment information from other images; Different fusion weights are reallocated and adjusted for the images to fuse into a single high dynamic range image.

2. The method of claim 1, wherein the plurality of images are aligned to one of the plurality of images by equalizing different exposure brightnesses based on a mutual gamma calculated based on image histograms and tracking features within the equalized images.

3. The method according to claim 2, characterized in that After brightness equalization of the aligned images using mutual gamma values ​​derived from mutual 2D histograms, non-similar regions in the images are found.

4. The method of claim 3, wherein each non-similar region is taken from its reference image, transformed to the other two exposures using mutual gamma, and copied to part or all of the image at the corresponding exposure.

5. The method according to claim 1, characterized in that The fusion weights within the non-similar regions are redistributed between the new local reference image for each region and the regions in the neighboring images that are similar to it but at different exposures.

6. The method of claim 5, wherein weights are adjusted after decomposition in a Gaussian pyramid to adjust global tone mapping and local contrast enhancement.

7. The method according to claim 1, characterized in that The method further includes: Frame alignment is performed on the image formed by the frames captured during the burst capture.

8. The method according to claim 7, characterized in that The performing frame alignment further comprises: Equalize the brightness of the image through corresponding mutual gamma correction; Detect the features to be tracked in the reference image and track the features in each of the other images; A homography transformation is performed to warp each image from the other images to match the reference image.

9. The method according to claim 1, characterized in that The method further includes: Seven input low-resolution images were captured at three exposure and sensitivity settings; The third and fifth images are converted to the pixel brightness values ​​of the fourth reference image by mutual gamma; Combine the two difference masks by an OR logic operation, and use connected component labeling to group adjacent non-similar pixels together to form different regions; The pixel values ​​of each region are analyzed to determine which exposure should be used to acquire each region.

10. An apparatus for combining low dynamic range images into a single image, comprising: An acquisition module, for acquiring low dynamic range images with different EV compensation values, including: image exposure and sensitivity settings are taken in a specific order: dark-normal-bright-normal-dark-normal-bright or bright-normal-dark-normal-bright-normal-dark; a detection module for detecting non-similar regions of the image by equalizing the image brightness according to a mutual two-dimensional histogram; A modification module is used to modify the image in the non-similar region before performing Laplace decomposition on the image, including: replacing the non-similar region with brightness adjustment information from other images; The fusion module is used to reallocate and adjust different fusion weights of images to fuse them into a single high dynamic range image.

11. An electronic device, comprising: processor; and a memory for storing processor executable instructions, The processor is configured to execute the method according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium storing executable instructions, which, when executed by an electronic device having a touch-sensitive display, causes the electronic device to perform the method of any one of claims 1-9.

Images