A two-exposure image fusion method based on guided filtering multi-level decomposition

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to a two-exposure image fusion method based on guided filtering multi-level decomposition. The present invention decomposes an image into a base layer and multiple detail layers through guided filtering, and then uses exposure weights and global gradient weights to mine the image information contained in the base layer and the detail layer to reconstruct the image. The present invention solves the problem of poor fused image quality when the original image sequence has a large exposure time difference in the prior art, and can adapt to a variety of exposure ratios and complex real scenes, and retains the overall brightness and local details through exposure weights and global gradient weights.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a two-exposure image fusion method based on guided filtering multi-level decomposition. Background Art

[0002] Dynamic range refers to the ratio of the maximum and minimum values ​​of a changing signal (such as sound or light). The dynamic range of brightness of a natural scene can exceed 100,000:1. Consumer-grade digital cameras have limited dynamic range due to manufacturing costs, and it is difficult to capture real natural scenes with high dynamic range (HDR) through a single imaging. There may be overexposed or underexposed areas in the image, resulting in loss of image details. In order to solve the above problems, multi-exposure fusion technology is currently widely used to generate high dynamic range images. First, multiple low dynamic range image sequences with different exposure times of the same scene are obtained, and then the image fusion method is used to reconstruct the image so that the newly generated image can retain the image details of both high brightness and low brightness areas. This method does not require professional imaging equipment and additional information of the camera, is universal and practical, and the generated new image has rich information to meet the authenticity of human visual perception.

[0003] Most existing multi-exposure fusion methods require a long sequence of original images with small exposure differences to obtain good fusion results. When there are fewer original images and they have large exposure differences, the quality of the fused image will be significantly reduced. In the 2018 paper "Multi-Scale Fusion of Two Large-Exposure-Ratio Images", Yang et al. proposed a method to generate a virtual image with medium exposure through an intensity mapping function, and then fuse the two original images and the virtual image to generate the final fused image. This method reduces the reliability of the fused image and also increases the computational complexity of the algorithm. Summary of the invention

[0004] The purpose of the present invention is to solve the problem of poor fused image quality when the original image sequence has a large exposure time difference, especially when the two input original images are under-exposed and over-exposed images respectively, the existing multi-exposure image fusion method cannot reconstruct the relative contrast information in the fused image well. The present invention proposes a two-exposure image fusion method based on guided filtering multi-level decomposition, which decomposes the image into a base layer and multiple detail layers through guided filtering, and then uses exposure weights and global gradient weights to mine the image information contained in the base layer and the detail layer to reconstruct the image.

[0005] A two-exposure image fusion method based on guided filtering multi-level decomposition includes the following steps:

[0006] Step 1: Obtain two original RGB channel images I(i) to be fused, and convert the two original RGB channel images I(i) into YCbCr channel images; where i=1,2;

[0007] Step 2: Obtain the exposure weight map of the YCbCr channel image in the Y channel, and obtain the exposure saliency weight W through Gaussian filtering. e (i);

[0008] Step 3: Use the original RGB channel image I(i) itself as the guide image to perform guided filtering to separate it into the base layer B1(i) and the detail layer D1(i);

[0009] Step 4: Continue to perform guided filtering on the base layer B1(i) obtained in step 3 using itself as the guide map to obtain B2(i) and D2(i). Similarly, continue to iterate to obtain B n (i) and D n (i) Decompose the image into:

[0010] I(i)=B1(i)+D1(i)

[0011] =B2(i)+D2(i)+D1(i)

[0012] =…

[0013] =B n (i)+D n (i)+D n-1 (i)+…++D1(i)

[0014] Let base layer B(i) = B n (i), the detail layer is D(i)=D n (i)+D n-1 (i)+…++D1(i);

[0015] Step 5: Frame by frame at each detail layer D n (i) Take the global gradient and process it through Gaussian filtering to get the significance weight W of each layer g,n (i);

[0016] Step 6: Calculate the base layer exposure weights of the fused image and the weights of the detail layers

[0017]

[0018]

[0019] Step 7: Based on the multi-level decomposition of the image and the weight maps at each level, image fusion is performed at the base layer B(i) and the detail layer D(i) to obtain a fused image F;

[0020]

[0021] Furthermore, the exposure significance weight W in step 2 e The calculation method of (i) is:

[0022]

[0023] Where σ is the variance; T1 and T2 are exposure thresholds; mean(I(i)) is the average pixel intensity value of the original RGB channel image I(i); and γ is a coefficient used to increase the robustness of the adaptation weight.

[0024] Furthermore, the guided filtering in step 3 is an edge-preserving filtering operation, and the loss function of the guided filtering based on the least squares method is as follows:

[0025]

[0026] in, are the local window ω centered at pixel k. k The linear transformation coefficient of i is the guiding image; P i is the original image; ε is the regularization parameter, ω, μ, δ are the number of pixels, mean and variance of the local window respectively; the final output sequence is:

[0027]

[0028] in, Gf(I,P,r,ε) represents the guided filtering operator, r is the filter radius, and after the guided filtering operation, the original image is divided into the base layer B1(i) and the detail layer D1(i):

[0029] B1(i)=Gf(I(i),I(i),r,ε), D1(i)=I(i)-B1(i).

[0030] Furthermore, in step 5, each detail layer D n (i) Take the global gradient and process it through Gaussian filtering to get the significance weight W of each layer g,n The method of (i) is specifically:

[0031] Each pixel in the RGB image sequence contains three components: red, green and blue. Each pixel is represented by a three-dimensional vector To express:

[0032]

[0033]

[0034] in, is a unit vector, R, G, B are the three components of red, green and blue; calculate the parameter g of the vector gradient xx , g yy , g xy They are:

[0035]

[0036]

[0037]

[0038] The gradient direction angle θ is:

[0039]

[0040] The gradient Grad(θ) is:

[0041]

[0042] The beneficial effects of the present invention are:

[0043] The present invention proposes a two-exposure image fusion method based on guided filtering multi-level decomposition, which decomposes the image into a base layer and multiple detail layers through guided filtering, and then uses exposure weights and global gradient weights to mine the image information contained in the base layer and the detail layer to reconstruct the image fusion strategy. The present invention solves the problem of poor fused image quality when the original image sequence has a large exposure time difference in the prior art, can adapt to a variety of exposure ratios and complex real scenes, and retains the overall brightness and local details through exposure weights and global gradient weights. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is the overall flow chart of the present invention.

[0045] Figure 2 This is a flowchart of multi-level image decomposition based on guided filtering in the present invention.

[0046] Figure 3 This is a flowchart of the multi-level image fusion in the present invention.

[0047] Figure 4 This is a comparison chart of the multi-exposure fusion structural similarity (MEF-SSIM) results of the present invention and nine multi-exposure fusion technologies. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with the accompanying drawings.

[0049] The purpose of the present invention is to solve the problem of poor fused image quality when the original image sequence has a large exposure time difference, especially when the two input original images are under-exposed and over-exposed images respectively, the existing multi-exposure image fusion method cannot reconstruct the relative contrast information in the fused image well. The present invention proposes a two-exposure image fusion method based on guided filtering multi-level decomposition, which decomposes the image into a base layer and multiple detail layers through guided filtering, and then uses exposure weights and global gradient weights to mine the image information contained in the base layer and the detail layer step by step to reconstruct the image fusion strategy. The method can adapt to a variety of exposure ratios and complex real scenes, and compared with the existing algorithms, it has achieved better fusion results.

[0050] The object of the present invention is achieved as follows: first, the guided filter with the original image as the guide decomposes the image to be fused into the first base layer and the detail layer, and then the guided filter with the base layer image as the guide continues to decompose the base layer into the second base layer and the detail layer, and so on, the original image can be decomposed into a base layer and multiple detail layers. Finally, the image information contained in the base layer and the detail layer is mined using the exposure weight and the global gradient weight to reconstruct the image.

[0051] The main idea of ​​this new two-exposure image fusion method is to use guided filtering to decompose the image into a base layer and a detail layer step by step, and then use the exposure weight and the global gradient weight map to mine the base layer and detail layer information of the original image at multiple levels.

[0052] The working principle of the two-exposure fusion based on multi-level decomposition of guided filtering image disclosed in the present invention is as follows: Figure 1 As shown, it mainly includes the following steps:

[0053] Step 1. Convert the original RGB channel image I(i) (i=1,2) into a YCbCr channel image;

[0054] Step 2. Get the exposure weight of each frame in the Y channel (i.e., brightness channel), and obtain the saliency weight map W after Gaussian filtering. e (i), where the saliency weight W is used as the base layer e (i) A solution for adapting the weight factor is proposed.

[0055]

[0056] Where σ is the variance, and T1 and T2 are exposure thresholds. In order to make the fused image reflect the details of the poor exposure sequence as much as possible, we need to understand the details of the low pixel intensity positions in the underexposed sequence and the high pixel intensity positions in the overexposed sequence. To this end, an adaptation factor of 1-mean(I(i)) is introduced, where mean(I(i)) is the average pixel intensity value of image I(i). In addition, a coefficient γ is added to increase the robustness of the adaptation weight. This parameter can be adjusted according to the exposure size of the specific scene and exposure sequence.

[0057] Step 3. Use the original image RGB channel image itself as the guide map to perform guided filtering to decompose the image. Among them, guided filtering is a more popular edge-preserving filtering operation, and the loss function of the guided filtering based on the least squares method is as follows:

[0058]

[0059] in,

[0060] are the local window ω centered at pixel k. k The linear transformation coefficient, I i is the guiding image, P i is the original image, ε is the regularization parameter, ω, μ, δ are the number of pixels, mean and variance of the local window respectively, and the final output sequence is:

[0061]

[0062] in, In this paper, Gf(I,P,r,ε) represents the guided filtering operator, and r is the filter radius. After the guided filtering operation, the original image is divided into the base layer B1(i) and the detail layer D1(i), that is,

[0063] B1(i)=Gf(I(i),I(i),r,ε), D1(i)=I(i)-B1(i);

[0064] Step 4. The base layer B1(i) obtained in step 3 is further subjected to guided filtering using itself as the guide map to obtain B2(i) and D2(i), that is,

[0065] B2(i)=Gf(B1(i),B1(i),r,ε), D2(i)=B1(i)-B2(i)

[0066] And similarly, we can iterate and get B n (i) and D n (i), decompose the image into

[0067] I(i)=B1(i)+D1(i)

[0068] =B2(i)+D2(i)+D1(i)

[0069] =…

[0070] =B n (i)+D n (i)+D n-1 (i)+…++D1(i)

[0071] Let base layer B(i) = B n (i), the detail layer is D(i)=D n (i)+D n-1 (i)+…++D1(i), such as Figure 2 As shown, two weights are designed to mine the image information of the base layer and the detail layer respectively;

[0072] Step 5. D n (i) Take the global gradient and process it through Gaussian filtering to get the weight map W of each level of saliency edge g,n (i) The main steps of calculating the global gradient are as follows: Each pixel in the RGB image sequence contains three components: red, green, and blue. Thus, each pixel can be represented by a three-dimensional vector To express:

[0073]

[0074]

[0075] in, is a unit vector, R, G, B are the three components of red, green and blue. Then the parameters of the vector gradient are calculated as g xx , g yy , g xy :

[0076]

[0077]

[0078]

[0079] Gradient direction angle θ:

[0080]

[0081] The gradient Grad(θ) is:

[0082]

[0083] The above formula can be used to take the gradient at each level of detail layer frame by frame as the global gradient weight Wg,n (i) It can fully explore the detailed information of under-exposed and over-exposed images;

[0084] Step 6. Base layer exposure weighting and the weights of the detail layers as follows,

[0085]

[0086]

[0087] in, is the result of the exposure weights being normalized and filtered, It is the result of normalizing and filtering the global gradient weight of the n-th level detail layer.

[0088] Step 7. Based on the above-obtained multi-level image decomposition and the weight maps at each level, image fusion is performed at the base layer B(i) and the detail layer D(i) to obtain a fused image F, as shown in Figure 3 As shown, that is:

[0089]

[0090] Finally, the multi-exposure fusion structural similarity (MEF-SSIM) results are used as a criterion to measure the performance of the algorithm.

[0091] like Figure 4 The figure shows the comparison of the multi-exposure fusion structural similarity (MEF-SSIM) results of the present invention and nine multi-exposure fusion technologies. 39 different two-exposure image sequences were selected, and the algorithm was tested and analyzed from both subjective and objective aspects. The experimental results show that the present invention can better adapt to different exposure conditions of the original image, and retain the overall brightness and local details through exposure weights and global gradient weights respectively. Compared with the prior art, the present invention is superior to the other nine methods in objective evaluation, and no color distortion and fusion artifacts appear in all test image scenes.

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A two-exposure image fusion method based on guided filtering multi-level decomposition, characterized in that: The following steps are involved: Step 1: Obtain two original RGB channel images I(i) to be fused, and convert the two original RGB channel images I(i) into YCbCr channel images; where i=1,2; Step 2: Obtain the exposure weight map of the YCbCr channel image in the Y channel, and obtain the exposure saliency weight W through Gaussian filtering. e (i); Step 3: Use the original RGB channel image I(i) itself as the guide image to perform guided filtering to separate it into the base layer B1(i) and the detail layer D1(i); Step 4: Continue to perform guided filtering on the base layer B1(i) obtained in step 3 using itself as the guide map to obtain B2(i) and D2(i). Similarly, continue to iterate to obtain B n (i) and D n (i), decompose the image into: I(i)=B1(i)+D1(i) =B2(i)+D2(i)+D1(i) ＝... =B n (i)+D n (i)+D n-1 (i)+…++D1(i) Let base layer B(i) = B n (i), the detail layer is D(i)=D n (i)+D n-1 (i)+…++D1(i); Step 5: Frame by frame at each detail layer D n (i) Take the global gradient and process it through Gaussian filtering to get the significance weight W of each layer g,n (i); Step 6: Calculate the base layer exposure weights of the fused image and the weight of the detail layer Step 7: Based on the multi-level decomposition of the image and the weight maps at each level, image fusion is performed at the base layer B(i) and the detail layer D(i) to obtain a fused image F; 2. The two-exposure image fusion method based on guided filtering multi-level decomposition according to claim 1, characterized in that: The exposure significance weight W in step 2 e The calculation method of (i) is: Where σ is the variance; T1 and T2 are exposure thresholds; mean(I(i)) is the average pixel intensity value of the original RGB channel image I(i); and γ is a coefficient used to increase the robustness of the adaptation weight.

3. The two-exposure image fusion method based on guided filtering multi-level decomposition according to claim 1, characterized in that: The guided filtering in step 3 is an edge-preserving filtering operation, and the loss function of the guided filtering based on the least squares method is as follows: in, are the local window ω centered at pixel k. k The linear transformation coefficient of i is the guiding image; P i is the original image; ε is the regularization parameter, ω, μ, δ are the number of pixels, mean and variance of the local window respectively; the final output sequence is: in, Gf(I,P,r,)) represents the guided filtering operator, r is the filter radius, and after the guided filtering operation, the original image is divided into the base layer B1(i) and the detail layer D1(i): B1(i)=Gf(I(i),I(i),r,ε), D1(i)=I(i)-B1(i).

4. The two-exposure image fusion method based on guided filtering multi-level decomposition according to claim 1, characterized in that: In step 5, each detail layer D is n (i) Take the global gradient and process it through Gaussian filtering to get the significance weight W of each layer g,n The method of (i) is specifically: Each pixel in the RGB image sequence contains three components: red, green and blue. Each pixel is represented by a three-dimensional vector To express: in, is a unit vector, R, G, B are the three color components of red, green and blue; calculate the parameter g of the vector gradient xx , g yy , g xy They are: The gradient direction angle θ is: The gradient Grad(θ) is:

Images