A multi-exposure fusion method separating illumination and motion

Abstract

The application belongs to the technical field of image processing, and particularly relates to a multi-exposure fusion method for separating illumination and motion; low-exposure images, medium-exposure images and high-exposure images are subjected to illumination preprocessing and input into a fusion model; the low-exposure images and the high-exposure images are mapped to the illumination level of the medium-exposure images and normalized according to first gamma values and second gamma values, to obtain first new medium-exposure images and second new medium-exposure images, and the new medium-exposure images are subjected to motion alignment with the medium-exposure images; the first new medium-exposure images and the second new medium-exposure images are subjected to inverse gamma transformation respectively, to obtain new low-exposure images and new high-exposure images, and the new low-exposure images and the new high-exposure images are subjected to illumination fusion with the medium-exposure images to obtain a high dynamic range image, loss is calculated, and model parameters are trained by back propagation; a to-be-processed image is obtained, and a corresponding high dynamic range image of the to-be-processed image is obtained by using the trained fusion model; the application adopts the idea of separating illumination and motion, so that the high dynamic range image retains more information of the input image, and ghosting is reduced.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to a high dynamic range imaging method, specifically a multi-exposure fusion method based on the concept of separating illumination and motion. Background Technology

[0002] In the field of imaging, dynamic range refers to the range of light intensity that can be captured in a scene. Ordinary digital cameras typically cannot capture the entire dynamic range of a natural scene; images taken by these devices are called low dynamic range (LVR) images. Custom-designed capture devices can obtain images with a wider dynamic range, but they are expensive and bulky. Therefore, many high dynamic range (HDR) imaging methods have been proposed over the past few decades. Among these, multi-exposure fusion is a mainstream research direction.

[0003] Multi-exposure fusion combines multiple images of the same scene with different exposures to obtain a high dynamic range (HDR) image. Typically, this involves three images with different exposures: a low-exposure image, a medium-exposure image, and a high-exposure image. The fused HDR image contains most of the useful information from the three low-exposure images. If the three images are completely identical static images, illumination fusion problems arise, resulting in incomplete or incorrect extraction of lighting details in certain parts of the resulting image. If the three images contain moving objects or camera shake, ghosting problems occur, distorting the moving areas of the resulting image. After decades of effort, purely static multi-exposure fusion methods have achieved excellent results, but various problems arise when processing dynamic images. To address these issues, researchers have proposed many effective methods, from early optical flow alignment to the mainstream deep learning-based multi-exposure fusion methods of recent years. With the acceleration of devices such as GPUs, HDR images can be obtained quickly and well. The performance of multi-exposure fusion imaging technology is constantly improving, and the quality of the fused HDR images has been enhanced. However, the problems of light blending and ghosting have not been well resolved, and the most advanced multi-exposure blending methods still have a large number of problem areas. Summary of the Invention

[0004] To address the aforementioned issues and obtain high-quality high dynamic range images, this invention provides a multi-exposure fusion method that separates motion and illumination. This method aims to further improve the solution to the illumination fusion and ghosting problems, and to generate high dynamic range images more efficiently, clearly, and interpretably through multi-exposure fusion.

[0005] A multi-exposure fusion method that separates illumination and motion includes the following steps:

[0006] S1. Obtain multiple sets of image pairs to form a training set. Each set of image pairs consists of a low-exposure image, a medium-exposure image, and a high-exposure image of the same scene; and construct a fusion model.

[0007] S2. Perform illumination preprocessing on all images in any image pair;

[0008] S3. Input the image pair after illumination preprocessing into the fusion model, calculate the first gamma value of the low exposure image mapped to the medium exposure image, and the second gamma value of the high exposure image mapped to the medium exposure image;

[0009] S4. Map the illumination level of the low-exposure image to the illumination level of the medium-exposure image according to the first gamma value and normalize it to obtain the first new medium-exposure image;

[0010] S5. Map the illumination level of the high-exposure image to the illumination level of the medium-exposure image based on the second gamma value and normalize it to obtain the second new medium-exposure image;

[0011] S6. Align the motion regions of the medium exposure image, the first new medium exposure image, and the second new medium exposure image at the same illumination level;

[0012] S7. The aligned first and second new medium exposure images are respectively converted back to their original illumination levels by gamma inverse transform to obtain the new low exposure image and the new high exposure image.

[0013] S8. Perform illumination fusion on the medium-exposure image, the new low-exposure image, and the new high-exposure image to obtain a high-quality high dynamic range image. Calculate the loss and backpropagate to train the model parameters.

[0014] S9. Obtain the image to be processed and use the trained fusion model to obtain its corresponding high dynamic range image.

[0015] Furthermore, in step S3, the process of calculating the first gamma value mapped from the low-exposure image to the medium-exposure image is as follows:

[0016] S31. Calculate the gamma value of the low-exposure image mapped to the medium-exposure image by improving the gamma mapping formula, and obtain the mapping result of the low-exposure image based on the gamma value;

[0017] S32. Calculate the mean and variance of the mapping result and the medium-exposure image;

[0018] S33. Repeat steps S31-S32 multiple times, and select the gamma value corresponding to the minimum mean variance as the first gamma value.

[0019] Furthermore, the process of calculating the first gamma value in step S3 is the same as the process of calculating the second gamma value, wherein the improved gamma mapping formula is expressed as:

[0020]

[0021] Where H represents the image after gamma mapping of a low-exposure image or a high-exposure image, I represents the numerical matrix of the low-exposure image or the high-exposure image, and γ represents the gamma value.

[0022] Furthermore, step S6 uses the feature extraction method color-FastICA to align the moving regions.

[0023] Furthermore, in step S8, an iterative fusion method based on multi-scale edge sharpening is selected for illumination fusion.

[0024] Furthermore, the loss during training consists of two parts: the first part is the L1 loss between the training image and its fused image, and the second part is the brightness loss between the training image and its fused image; the weight ratio of the two parts of the loss is 1:1.

[0025] Furthermore, the brightness of the image is calculated using the RGB brightness formula, expressed as:

[0026] Brightness = 0.3 * R + 0.6 * G + 0.1 * B

[0027] R, G, and B are the three color channels of RGB.

[0028] The beneficial effects of this invention are:

[0029] This invention fully considers the ghosting problem and dynamic part alignment problem in multi-exposure fusion. It adopts the idea of ​​separating illumination and motion to address these two problems in a targeted manner, so that the final high dynamic range image can retain more information from the three input images and less ghosting, resulting in a high-quality high dynamic range image. Attached Figure Description

[0030] Figure 1 This is a flowchart of a multi-exposure fusion method for separating illumination and motion according to the present invention;

[0031] Figure 2 This is a set of image pairs in one embodiment of the present invention;

[0032] Figure 3 This is an image obtained by gamma mapping of a low-exposure image and a high-exposure image in one embodiment of the present invention;

[0033] Figure 4 This is a high dynamic range diagram of the fusion result according to an embodiment of the present invention;

[0034] Figure 5 These are low-exposure and medium-exposure images in one embodiment of the present invention;

[0035] Figure 6 This is an image obtained by mapping a low-exposure image to a medium-exposure image in one embodiment of the present invention;

[0036] Figure 7 This is a comparison diagram of the alignment effect of illumination levels in one embodiment of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] To achieve the effect of separating light and motion, there are three main directions to try:

[0039] The first direction is to iterate on lighting level adjustment and motion alignment processing.

[0040] The second approach involves performing motion alignment first, then illumination transformation, and finally merging. If the moving areas can be perfectly aligned, then it becomes a static multi-exposure fusion problem, which can be solved using previous static multi-exposure fusion methods. However, the difficulty of this approach lies in aligning the moving areas; achieving the effect of aligning the moving areas to a static image is virtually impossible.

[0041] The third approach involves first performing illumination transformation, then alignment, and finally merging. This method first adjusts the illumination of the multi-exposure images to the same level, then performs motion alignment. Motion alignment under the same illumination conditions yields significantly better results than under different illumination conditions, thus significantly reducing ghosting issues in the final merged image. However, this approach presents challenges in adjusting the illumination level and performing motion alignment.

[0042] This invention proposes a multi-exposure fusion method that separates illumination and motion, addressing a third-direction idea. It first performs illumination transformation, then alignment processing, and finally fusion. For example... Figure 1 As shown, it specifically includes:

[0043] S1. Obtain multiple sets of image pairs to form a training set. Each set of image pairs consists of a low-exposure image, a medium-exposure image, and a high-exposure image of the same scene, such as... Figure 2 As shown; and a fusion model is constructed;

[0044] S2. Perform illumination preprocessing on all images in any image pair;

[0045] Specifically, gamma correction is used to preprocess the illumination of each image pair, so that the dynamic range of each image is no longer limited to 0-25, providing richer feature details and a wider dynamic range for subsequent feature extraction.

[0046] S3. Input the image pair after illumination preprocessing into the fusion model, calculate the first gamma value of the low exposure image mapped to the medium exposure image, and the second gamma value of the high exposure image mapped to the medium exposure image;

[0047] S4. Map the illumination level of the low-exposure image to the illumination level of the medium-exposure image according to the first gamma value and normalize it to obtain the first new medium-exposure image;

[0048] S5. Map the illumination level of the high-exposure image to the illumination level of the medium-exposure image based on the second gamma value and normalize it to obtain the second new medium-exposure image;

[0049] S6. Align the motion regions of the medium exposure image, the first new medium exposure image, and the second new medium exposure image at the same illumination level;

[0050] S7. The aligned first and second new medium exposure images are respectively converted back to their original illumination levels by gamma inverse transform to obtain the new low exposure image and the new high exposure image.

[0051] S8. Perform illumination fusion on the medium-exposure image, the new low-exposure image, and the new high-exposure image to obtain a high-quality, high dynamic range image, such as... Figure 4 As shown, the loss is calculated and the model parameters are trained by backpropagation;

[0052] S9. Obtain the image to be processed and use the trained fusion model to obtain its corresponding high dynamic range image.

[0053] Specifically, in step S3, the Gamma value is estimated for the three multi-exposure images. This can be done by combining MSE constraints or average pixel constraints. In this embodiment, MSE constraints are chosen.

[0054] Specifically, the process of calculating the first gamma value of the low-exposure image mapped to the medium-exposure image is as follows:

[0055] S301. Calculate the gamma value of the low-exposure image mapped to the medium-exposure image by improving the gamma mapping formula, and obtain the mapping result of the low-exposure image based on the gamma value;

[0056] S302. Calculate the mean and variance of the mapping result and the medium-exposure image;

[0057] S303. Repeat steps S301-S302 multiple times, and select the gamma value corresponding to the minimum mean variance as the first gamma value.

[0058] Specifically, the process of calculating the second gamma value of the high-exposure image mapped to the medium-exposure image is as follows:

[0059] S311. Calculate the gamma value of the high-exposure image mapped to the medium-exposure image by improving the gamma mapping formula, and obtain the mapping result of the high-exposure image based on the gamma value;

[0060] S312. Calculate the mean and variance of the mapping result and the medium-exposure image;

[0061] S313. Repeat steps S311-S312 multiple times, and select the gamma value corresponding to the minimum mean variance as the second gamma value.

[0062] The gamma correction formula in the high dynamic range imaging direction is usually H=I. γ / t, where t is the image exposure time, which varies for images with different exposures. This invention dynamically adjusts the illumination level, requiring no explicit exposure time, and the exposure times in the dataset are not entirely accurate. Furthermore, the effect of changing the illumination level can be achieved without introducing t, and it can be viewed as a power function, capable of inverse transformation. Therefore, this invention uses an improved gamma mapping formula, expressed as:

[0063]

[0064] Where H represents the image after gamma mapping of a low-exposure image or a high-exposure image, I represents the numerical matrix of the low-exposure image or the high-exposure image, and γ represents the gamma value.

[0065] Secondly, find the gamma value corresponding to the minimum mean-variance. Specifically, take the low-exposure image (or high-exposure image) after gamma transformation with a certain gamma value and the medium-exposure image, and calculate the minimum mean-variance. Different gamma values ​​have different minimum means, which can form a quadratic function. Find the minimum point of this quadratic function, which is the optimal gamma value. The gamma value variation interval is 0.01.

[0066] Gamma transformation combined with minimum mean variance is necessary for effective lighting adjustment.

[0067] Specifically, such as Figure 3 As shown, Figure 3 (a) has a Gamma value of 1.15 and an MSE of 1922 at the time of generation. Figure 3 (b) has a Gamma value of 0.91 and an MSE of 1734 at the time of generation.

[0068] Specifically, step S6 uses the color-FastICA feature extraction method for motion region alignment. Color-FastICA is a fast iterative optimization algorithm that uses batch processing and iteratively optimized ICA algorithm to separate image signals from consecutive exposure sequences. It can separate the difference features of multi-exposure images in the RGB channels, such as structural and color detail differences, and finally obtain the feature extraction results of the prominent region, with values ​​in the range [0,1].

[0069] Specifically, the loss during the training process of this invention consists of two parts: the first part is the L1 loss between the training image and its fused image, and the second part is the brightness loss between the training image and its fused image; the weight ratio of the two parts is 1:1, expressed as:

[0070] ;

[0071]

[0072] Where L represents the total loss function, L1 represents the first part of the training loss, and L2 represents the second part of the training loss. This indicates the brightness of the original high-exposure image. Brightness represents the brightness of a high-quality, high dynamic range image. This indicates the brightness of the original low-exposure image. This indicates the brightness of the exposed image.

[0073] The main purpose of the second part of the loss is:

[0074] 1. The illumination level of the medium-exposure image serves as a reference for the illumination level of the target image. Therefore, the brightness loss of the fused image (high-quality high dynamic range image) is calculated by using the illumination level of the medium-exposure image as the target.

[0075] 2. Minimize the brightness difference between input images by making subtle adjustments to the brightness of the model modules for the low-exposure image input channel and the high-exposure image input channel, so as to blend the low-exposure and high-exposure images with the medium-exposure image at a similar illumination level.

[0076] Before the final fusion, a luminance loss was added to adjust for subtle differences in image illumination levels, optimizing the final fusion result. Simple ablation experiments showed that before and after adding the luminance loss, PSNR-L increased by 2.0, and PSNR-μ increased by 3.1, both showing significant improvements.

[0077] Ablation experiments show that motion alignment of input images at the same illumination level yields better final fusion results than motion alignment at different illumination levels, with improvements in metrics such as VIFP, ILNIQE, PSNR, and UQI. (VIFP is an image quality assessment parameter based on the concept of image signals extracted from natural scene statistics and the human visual system; ILNIQE is a fully blind, distortion-free, and reference-standard-free image quality assessment metric; Peak Signal-to-Noise Ratio (PSNR) is the most commonly used and widely applied objective image quality assessment metric, based on the error between pixels; UQI uses correlation loss, brightness distortion, and contrast distortion to evaluate image quality.)

[0078] Furthermore, this invention also conducted experiments using optical flow alignment, finding that the aligned image of two images at the same illumination level had a smoother and brighter optical flow field and lower optical flux than images at different illumination levels. This further proves that alignment at the same illumination level has a greater effect and lower energy consumption than alignment at different illumination levels. Figure 5 , Figure 6 As shown, where Figure 5 (a) is a low-exposure image. Figure 5 (b) is a medium exposure image. Figure 6 This is an image showing the effect of changing the illumination level of a low-exposure image to that of a medium-exposure image using the method presented in this paper. Then, optical flow is applied to... Figure 5 (a) and Figure 5 (b) Figure 6 and Figure 5 (b) Align the results to obtain... Figure 7 (a) and Figure 7 (b) The effect diagram shows that, visually, the optical flow field of the second group is significantly smoother and brighter, with less optical flux. In terms of indicators, the maximum optical flux of the first group is 194.0116, while that of the second group is 158.1365, which is significantly lower.

[0079] Specifically, step S8 selects an iterative fusion method based on multi-scale edge sharpening for illumination fusion. First, the original image (referring to the medium-exposure image, the new low-exposure image, or the new high-exposure image) is processed with Gaussian kernels of different scales to obtain images with different degrees of blur. Then, a simple difference calculation is performed between the original image and the upsampled image to obtain different degrees of texture detail. Finally, the final fused image is obtained by integrating them with different weights.

[0080] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "setting," "connection," "fixing," "rotation," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Unless otherwise explicitly limited, those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0081] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-exposure fusion method that separates illumination and motion, characterized in that, Includes the following steps: S1. Obtain multiple sets of image pairs to form a training set. Each set of image pairs consists of a low-exposure image, a medium-exposure image, and a high-exposure image of the same scene; and construct a fusion model. S2. Perform illumination preprocessing on all images in any image pair; S3. Input the image pair after illumination preprocessing into the fusion model, calculate the first gamma value of the low exposure image mapped to the medium exposure image, and the second gamma value of the high exposure image mapped to the medium exposure image; The process of calculating the first gamma value of a low-exposure image mapped to a medium-exposure image is as follows: S301. Calculate the gamma value of the low-exposure image mapped to the medium-exposure image by improving the gamma mapping formula, and obtain the mapping result of the low-exposure image based on the gamma value; S302. Calculate the mean and variance of the mapping result and the medium-exposure image; S303. Repeat steps S301-S302 multiple times, and select the gamma value corresponding to the minimum mean variance as the first gamma value; The process of calculating the second gamma value of a high-exposure image mapped to a medium-exposure image is as follows: S311. Calculate the gamma value of the high-exposure image mapped to the medium-exposure image by improving the gamma mapping formula, and obtain the mapping result of the high-exposure image based on the gamma value; S312. Calculate the mean and variance of the mapping result and the medium-exposure image; S313. Repeat steps S311-S312 multiple times, and select the gamma value corresponding to the minimum mean variance as the second gamma value; The improved gamma mapping formula is expressed as: Where H represents the image after gamma mapping of a low-exposure image or a high-exposure image, I represents the numerical matrix of the low-exposure image or the high-exposure image, and γ represents the gamma value. S4. Map the illumination level of the low-exposure image to the illumination level of the medium-exposure image according to the first gamma value and normalize it to obtain the first new medium-exposure image; S5. Map the illumination level of the high-exposure image to the illumination level of the medium-exposure image based on the second gamma value and normalize it to obtain the second new medium-exposure image; S6. Align the motion regions of the medium exposure image, the first new medium exposure image, and the second new medium exposure image at the same illumination level; S7. The aligned first and second new medium exposure images are respectively converted back to their original illumination levels by gamma inverse transform to obtain the new low exposure image and the new high exposure image. S8. Perform illumination fusion on the medium-exposure image, the new low-exposure image, and the new high-exposure image to obtain a high-quality high dynamic range image. Calculate the loss and backpropagate to train the model parameters. S9. Obtain the image to be processed and use the trained fusion model to obtain its corresponding high dynamic range image.

2. The multi-exposure fusion method for separating illumination and motion according to claim 1, characterized in that, Step S6 uses the feature extraction method color-FastICA to align the moving regions.

3. The multi-exposure fusion method for separating illumination and motion according to claim 1, characterized in that, Step S8 selects an iterative fusion method based on multi-scale edge sharpening for illumination fusion.

4. The multi-exposure fusion method for separating illumination and motion according to claim 1, characterized in that, The loss during training consists of two parts: the first part is the L1 loss between the training image and its fused image, and the second part is the brightness loss between the training image and its fused image; the weight ratio of the two parts is 1:

1.

5. The multi-exposure fusion method for separating illumination and motion according to claim 4, characterized in that, The brightness of an image is calculated using the RGB brightness formula, expressed as: Brightness = 0.3 * R + 0.6 * G + 0.1 * B R, G, and B are the three color channels of RGB.

Images