Adaptive weight detail preserving multi-exposure image fusion algorithm

Abstract

The application relates to a self-adaptive weight detail keeping multi-exposure image fusion algorithm, which comprises the following steps: acquiring multi-exposure images to be fused; calculating the exposure weight of each image in the multi-exposure image group; calculating the structure weight of each image in the multi-exposure image group; calculating the saturation weight of each image in the multi-exposure image group; calculating a fusion weight map, carrying out guided filtering on the weight map and then normalizing; and carrying out double-pyramid decomposition and fusion on the source image sequence and the filtered weight map. In the application, the contrast and structure components in the image structure block decomposition are used for calculating the pixel fusion weight, greater weight is given to the pixel area with strong contrast and structure, and the rich image details in the source image sequence are kept; the brightness weight is based on the image brightness at the maximum two-dimensional entropy and is combined with the saturation weight, so that the fused image can better restore the brightness and color information of the scene; and the double-pyramid fusion fuses the source image sequence on multiple scales to avoid unnatural halos at the boundaries.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image fusion, and particularly relates to a self-adaptive weight detail preserving multi-exposure image fusion algorithm. BACKGROUND

[0002] Natural scenes have a very high dynamic range, from moonless night to direct sunlight, the dynamic range is about 240dB, while the dynamic range of ordinary image sensors is usually about 70dB due to the limitation of the maximum potential well capacity. Therefore, it is difficult to collect complete scene information, resulting in information loss.

[0003] Current high dynamic range imaging technologies basically include three categories: sensor-based, light intensity modulation-based and image processing-based. Among them, the sensor-based and light intensity modulation-based methods belong to the hardware level, which are difficult to manufacture and expensive; while the image processing-based method acts on the software level, which is more flexible and efficient compared with the hardware method. The high dynamic range technology based on image processing includes tone mapping and multi-exposure image fusion. Tone mapping needs a lot of work in the early stage to obtain the camera response curve, which is time-consuming, while multi-exposure image fusion greatly reduces the workload and can provide more scene details compared with tone mapping.

[0004] Multi-exposure fusion technology has developed from pixel-based fusion to image block-based fusion. The traditional pixel-based fusion technology takes 0.5 luminance value as the benchmark, and the fusion contrast is poor. The image block-based fusion can better preserve image details and improve contrast compared with the pixel-based fusion, but it is easy to produce halo phenomenon at the boundary of the fused image, which does not conform to the human visual perception. SUMMARY

[0005] The application aims to solve the technical problems in the prior art, and provides a self-adaptive weight detail preserving multi-exposure image fusion algorithm.

[0006] In order to solve the above technical problems, the technical scheme of the application is as follows:

[0007] A self-adaptive weight detail preserving multi-exposure image fusion algorithm comprises the following steps:

[0008] Step 1: obtaining multi-exposure images to be fused;

[0009] In the same high dynamic range scene, the same device is used to take pictures at different exposure times to obtain N images with different exposure degrees. After registration of the N images, the multi-exposure images to be fused are obtained, and N is greater than or equal to 2;

[0010] Step 2: calculating the exposure weight E of each image in the multi-exposure image group;

[0011] The exposure of a single pixel is represented by the brightness value L in the YUV channel of the image, and the exposure quality E is evaluated using a Gaussian model function, expressed as:

[0012]

[0013] Where μ(i,j) represents the three-channel pixel mean at pixel (i,j) in the nth source image, exp represents the natural exponential function, and L n (i,j) represents the brightness value of the corresponding point in the image, σ is the Gaussian standard deviation, and E n (i,j) represents the exposure weight of the source image;

[0014] Step 3: Calculate the structure weight C of each image in the multi-exposure image group;

[0015] Using c in image local block decomposition n s n The component-based structure weight C is calculated as follows:

[0016]

[0017] in, This represents the desired contrast component in the blended block. Represents the desired structural components in a local block of the fused image;

[0018] Step 4: Calculate the saturation weight S of each image in the multi-exposure image group;

[0019] The saturation weight S of each source image sequence to the fused image is determined by the mean of the standard deviations of the pixel values ​​in the R, G, and B channels of the source image sequence, and is expressed as:

[0020]

[0021] Among them, I nR I nG with I nB These represent the pixel values ​​(μ) of the R, G, and B channels of the image, respectively. n (i,j) represents the average value of the three channels of the nth source image at pixel (i,j);

[0022] Step 5: Calculate the fused weight graph, and normalize the weight graph after applying guided filtering;

[0023] Fusion Weight Graph P n Calculated using the following formula:

[0024] P n =E n ×S n ×C n ;

[0025] wherein E n , S n , C n respectively represent exposure weight, saturation weight and structure weight;

[0026] Step six: double pyramid decomposition and fusion of the source image sequence and the filtered weight map are performed;

[0027] The lth layer of Laplacian pyramid decomposition of image I is represented as L{I} l , the normalized weight map is represented as The fusion image R is calculated according to the following formula:

[0028]

[0029] wherein respectively represent Laplacian pyramid of the fusion image, Laplacian pyramid of the source image and Gaussian pyramid of the weight map.

[0030] In the above technical solution, in step two, the two-dimensional entropy at pixel point (i, j) is represented as H n (i, j);

[0031]

[0032]

[0033] wherein u represents the luminance value at pixel point (i, j); v represents the luminance average value of the neighborhood of WxW centered at (i, j); P(u, v) represents the probability of f(u, v) occurring;

[0034] The exposure weight E performs exposure comparison based on the luminance value μ at the maximum two-dimensional entropy among the N images, and the reference luminance value μ is defined as:

[0035] μ(i, j) = L nmax (i, j)

[0036] wherein nmax represents the image sequence number n value corresponding to the maximum two-dimensional entropy at corresponding pixel point (i, j) among the N images, and the nmax value is obtained through the following formula:

[0037]

[0038] wherein H nmax represents the maximum two-dimensional entropy value at pixel point (i, j), and the subscript is the image sequence number nmax corresponding to the maximum two-dimensional entropy at corresponding pixel point (i, j) among the N images.

[0039] In the technical solution, in step three, With Solved by the following formula:

[0040]

[0041]

[0042] Where ||·|| represents the two norm of the vector, The structural component contribution weight of each source image local block in the fusion image, max represents the maximum value in the data group, Indicates the expected fusion image block structure, s n Indicates the unit length structure vector, Indicates the image block removing the mean, n indicates the current image sequence index, and N indicates the number of source images to be fused.

[0043] In the technical solution, in step three, Indicates:

[0044]

[0045] Determine the p value through the structure consistency measure R between vectors,

[0046]

[0047] Where ||·|| represents the two norm of the vector, n represents the current image sequence index, N represents the number of images to be fused, and tan represents the tangent function, Indicates the image block removing the mean.

[0048] In the technical solution, in step four, μ n The calculation formula of (i,j) is:

[0049] μ n (i,j) = [I nR (i,j) + I nG (i,j) + I nB (i,j)] / 3.

[0050] In the technical solution, step five is specifically:

[0051] Take the source image sequence I n As the reference, guide filtering is performed on each corresponding weight map P n Output the guided weight map W n :

[0052] W n = F r,ε (P n , In );

[0053] wherein, F r,ε denotes the guided filter, r and epsilon are two constant parameters in the guided filter;

[0054] After filtering and reserving the weight details, the N weight maps are normalized, and the normalized weight maps are calculated by the following formula:

[0055]

[0056] wherein, tau is a non-zero small value, W n (i,j), respectively denote the weight map after the guided filtering and the normalized weight map.

[0057] The present application has the following beneficial effects:

[0058] The adaptive weight detail preserving multi-exposure image fusion algorithm of the present application uses the contrast and structure components in the image structure block decomposition to calculate the pixel fusion weight, gives greater weight to the pixel area with strong contrast and structure, and reserves the rich image details in the source image sequence; the brightness weight takes the image brightness at the maximum two-dimensional entropy as the reference, and combines the saturation weight, so that the fused image can better restore the brightness and color information of the scene; the double pyramid fusion fuses the source image sequence at multiple scales, and can avoid the unnatural halo at the boundary. BRIEF DESCRIPTION OF DRAWINGS

[0059] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0060] Figure 1 It is an implementation flowchart of the adaptive weight detail preserving multi-exposure image fusion algorithm of the present application.

[0061] Figure 2 It is a schematic diagram of a group of multi-exposure image sequences to be fused.

[0062] Figure 3 It is a schematic diagram of the result of fusing the image sequences in the application by using the adaptive weight detail preserving multi-exposure image fusion algorithm of the present application. Figure 2 DETAILED DESCRIPTION

[0063] In order to improve the image contrast and reserve more image details, the present application proposes an adaptive weight detail preserving multi-exposure image fusion algorithm.

[0064] ​The adaptive weight detail preserving multi-exposure image fusion algorithm of the application utilizes the decomposition and extraction features of the structure degree and contrast of the image block decomposition to calculate the image fusion weight, and combines the image brightness at the maximum two-dimensional entropy as the benchmark of the brightness weight, so that the fused image preserves the image details and the original scene brightness information.

[0065] The adaptive weight detail preserving multi-exposure image fusion algorithm of the application, as shown in the specific process as follows: Figure 1

[0066] Step one: obtaining the multi-exposure images to be fused, in the same high dynamic range scene, using the same device to take pictures at different exposure times to obtain N (N≥2) images with different exposure degrees, and after registering the N images, the multi-exposure images to be fused are obtained, as shown in Figure 2

[0067] Step two: calculating the exposure weight E of each image in the multi-exposure image group, the exposure of a single pixel is represented by the brightness value L in the YUV channel of the image, and the exposure quality E is evaluated by a Gaussian model function, which is represented as:

[0068]

[0069] Step three: calculating the structure weight C of each image in the multi-exposure image group, using the c n , s n component in the local block decomposition of the image to calculate the structure weight C in the application, which is represented as:

[0070]

[0071] Step four: calculating the saturation weight S of each image in the multi-exposure image group, using the mean value of the standard deviation of the pixel values of the R, G and B channels in the source image sequence to determine the saturation weight S of each source image sequence to the fused image, which is represented as:

[0072]

[0073] Step five: calculating the fusion weight map, normalizing the guided filter of the weight map, and the fusion weight map P n is calculated by the following formula:

[0074] P n =E n ×S n ×C n

[0075] Taking the source image sequence I n as the benchmark, referring to the following formula to calculate the corresponding weight map P n ​​Guided filtering is performed, r=4 and ε=0.2 in the example, and the output guided weight map W is obtained n ;

[0076] W n = F r,ε (P n , I n )

[0077] After filtering and preserving the weight details, N weight maps are normalized, τ=10 in the example -12 , and the normalized weight map is calculated according to the following formula:

[0078]

[0079] Step six: the source image sequence and the filtered weight map are subjected to bi-pyramid decomposition and fusion, the lth layer of the Laplacian pyramid decomposition of the image I is represented as L{I} l , and the Gaussian pyramid of the normalized weight map is represented as The fusion image R is calculated according to the following formula, and the fusion result is shown in Figure 3 ,

[0080]

[0081] The application will be further described in detail below with reference to the accompanying drawings.

[0082] The adaptive weight detail preserving multi-exposure image fusion algorithm of the application, as shown in Figure 1 , includes the following steps:

[0083] Step one: obtaining the multi-exposure images to be fused:

[0084] In the same high dynamic range scene, the same device is used to take pictures at different exposure times, N (N≥2) images with different exposure levels are obtained, and after registration of the N images, the multi-exposure images to be fused are obtained, as shown in Figure 2 ;

[0085] Step two: calculating the exposure weight E of each image in the multi-exposure image group:

[0086] The exposure of a single pixel is represented by the luminance value L in the YUV channel of the image, and the exposure quality E is evaluated by a Gaussian model function, represented as:

[0087]

[0088] wherein μ(i,j) represents the three-channel pixel mean value of the nth source image at the pixel point (i,j), exp represents the natural exponential function, and Ln (i,j) is the luminance value of the image corresponding point, σ is the Gaussian standard deviation, E n (i,j) is the exposure weight of the source image;

[0089] The two-dimensional entropy at the pixel point (i,j) is expressed as H n (i,j);

[0090]

[0091]

[0092] Wherein, u represents the luminance value at the pixel point (i,j); v represents the luminance mean value of the neighborhood with (i,j) as the center and WxW, W=9 in the present application; P(u,v) represents the probability of f(u,v) occurrence.

[0093] The exposure weight E is compared with the exposure based on the luminance value μ at the maximum two-dimensional entropy between the N images, and the reference luminance value μ is defined as:

[0094] μ(i,j) = L nmax (i,j)

[0095] Wherein, nmax represents the image sequence number n value corresponding to the maximum two-dimensional entropy at the corresponding pixel point (i,j) between the N images, and the nmax value is obtained according to the following formula:

[0096]

[0097] Wherein, H nmax represents the maximum two-dimensional entropy value at the pixel point (i,j), and the corresponding subscript is the image sequence number nmax corresponding to the maximum two-dimensional entropy at the corresponding pixel point (i,j) between the N images.

[0098] Step three: calculate the structure weight C of each image in the multi-exposure image group:

[0099] For the structure weight, the present application decomposes the source image sequence into local blocks, and the image block set extracted from the same spatial position in the N multi-exposure image sequence is expressed as x n , wherein all x n is an M-dimensional column vector, wherein M is the number of pixels in the image block, and the image block is decomposed:

[0100]

[0101] Wherein, ||·|| represents the two-norm of the vector, is the mean value of the pixel gray scale in the image block, is the image block after removing the mean value, c n , s n, l n respectively represent the contrast, structure and intensity components of the image patch x n .

[0102] The structure weight C in the present application only adopts the c n , s n component in the image local patch decomposition, and is expressed as:

[0103]

[0104] wherein, represents the expected contrast component in the fused patch, which is determined by the highest contrast in all source image local patches, represents the expected structure component in the fused image local patch, which represents the structure degree of all source image local patches, and is solved by the following formula:

[0105]

[0106]

[0107] wherein, ||·|| represents the two-norm of a vector, max represents taking the maximum value in the data group after it, represents the expected structure of the fused image patch, s n represents the unit length structure vector, represents the image patch removing the mean value, n represents the current image sequence index, and N represents the number of source images to be fused, is the structure component contribution weight of each source image local patch in the fused image, which increases with the increase of the structure degree, and is expressed as:

[0108]

[0109] wherein, p is an exponential parameter, the greater the value of p, the greater the weight given to the source image patch with greater structure degree, p=0 corresponds to the direct direction average of the structure vector; p=1 corresponds to the length-weighted direction average; p=2 corresponds to the energy-weighted direction average; p=∞ corresponds to selecting the direction corresponding to the patch with the maximum vector length in all image patches. The value of p is determined by the structure consistency measure R between vectors.

[0110]

[0111]

[0112] wherein, ||·|| represents the two-norm of a vector, n represents the current image sequence index, N represents the number of source images to be fused, and tan represents the tangent function, represents the pixel block removing the mean value.

[0113] Step four: calculate the saturation weight S of each image in the multi-exposure image group:

[0114] The present application uses the mean value of the standard deviation of the R, G, B channel pixel values in the source image sequence to determine the saturation weight S of each source image sequence to the fusion image, and the calculation of S is represented as:

[0115]

[0116] Wherein, I nR , I nG and I nB are the pixel values of the R, G, B channels of the image, μ n (i,j) represents the three-channel pixel mean value of the nth source image at the pixel point (i,j), and the calculation method is represented by the following formula:

[0117] μ n (i,j) = [I nR (i,j) + I nG (i,j) + I nB (i,j)] / 3

[0118] Step five: calculate the fusion weight map, and normalize the guided filter of the weight map:

[0119] The fusion weight map P n is calculated by the following formula:

[0120] P n = E n × S n × C n

[0121] Wherein, E n , S n , and C n represent the exposure weight, saturation weight, and structure weight, respectively.

[0122] With the source image sequence I n as the reference, the guided filter is performed on each corresponding weight map P n according to the following formula, and the guided weight map W n is output:

[0123] W n = F r,ε (P n , I n )

[0124] Wherein, F r,ε represents the guided filter, and r and ε are two constant parameters in the guided filter. In the present application, r = 4 and ε = 0.2.

[0125] After filtering and reserving the weight details, the N weight maps are normalized so that the sum at each pixel point (i, j) is 1, and the normalized weight map is represented as

[0126]

[0127] Wherein, τ is a non-zero infinitesimal value, avoiding the denominator zero condition, in the present application, τ = 10 is taken -12 ; W n (i, j), Respectively represent the weight map after guided filtering and the normalized weight map.

[0128] Step six: pyramid decomposition and fusion of the source image sequence and the filtered weight map:

[0129] The input image is decomposed into a Laplace pyramid, and the weight map is decomposed into a Gaussian pyramid, and then each level of the pyramid is mixed. The lth layer of the Laplace pyramid decomposition of the image I is represented as L{I} l , and the Gaussian pyramid of the normalized weight map is represented as The fusion image R is calculated according to the following formula.

[0130]

[0131] Wherein, Respectively represent the Laplace pyramid of the fusion image, the Laplace pyramid of the source image and the Gaussian pyramid of the weight map.

[0132] The fusion result is shown in Figure 3 .

[0133] The adaptive weight detail preservation multi-exposure image fusion algorithm of the present application uses the contrast and structure components in image structure block decomposition to calculate the pixel fusion weight, gives greater weight to the pixel area with strong contrast and structure, and preserves the rich image details in the source image sequence; the brightness weight takes the image brightness at the maximum two-dimensional entropy as the reference, and combines the saturation weight, so that the fused image can better restore the brightness and color information of the scene; the double pyramid fusion fuses the source image sequence at multiple scales, which can avoid unnatural halo at the boundary.

[0134] Obviously, the above embodiments are merely example for clearly illustrating but not limitation to the embodiments. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all the embodiments need not and can not be enumerated. The obvious changes or variations derived from the above description are still within the protection scope of the present application.

Claims

1. An adaptive weight detail preserving multi-exposure image fusion algorithm, characterized in that, The method comprises the following steps: Step one: obtaining a multi-exposure image to be fused; In the same high dynamic range scene, the same device is used to take pictures under different exposure times, N images with different exposure degrees are obtained, and the N images are registered to obtain a multi-exposure image to be fused, N≥2; Step two: calculating the exposure weight E of each image in the multi-exposure image group; The exposure of a single pixel is represented by the luminance value L in the YUV channel of the image, and the exposure quality E is evaluated by a Gaussian model function, which is represented as: wherein μ(i,j) represents the three-channel pixel mean value of the nth source image at pixel point (i,j), exp represents the natural exponential function, L n (i,j) is the luminance value of the image corresponding point, σ is the Gaussian standard deviation, E n (i,j) is the luminance value of the image corresponding point, σ is the Gaussian standard deviation, E Step three: calculating the structure weight C of each image in the multi-exposure image group; The structure weight C is calculated by using the c n , s n component in the image local block decomposition, and is expressed as: wherein, represents a desired contrast component in the fusion block, represents a desired structure component in the local block of the fusion image; Step four: calculating the saturation weight S of each image in the multi-exposure image group; The mean value of the standard deviation of the pixel values of the R, G, and B channels in the source image sequence is used to determine the saturation weight S of each source image sequence to the fused image, which is represented as: where I nR , I nG and I nB are the pixel values of the image R, G, B channels, respectively, and μ n (i,j) represents the three-channel pixel mean value of the nth source image at pixel point (i,j). Step five: calculating the fusion weight map, guiding filtering the weight map, and normalizing; Fusion weight map P n is calculated by the following equation: P n = E n × S n × C n ; wherein E n , S n , C n represent the exposure weight, the saturation weight and the structure weight, respectively; Step six: double pyramid decomposition and fusion of the source image sequence and the filtered weight map; The lth level of the Laplacian pyramid decomposition of the image I is denoted as L{I} l , the normalized weight map The Gaussian pyramid of the image I is denoted as The fused image R is computed according to the following equation: wherein, respectively denote a Laplacian pyramid of the fused image, a Laplacian pyramid of the source image, and a Gaussian pyramid of the weight map. In step two, the two-dimensional entropy at pixel point (i,j) is represented as H n (i,j); Wherein, u represents the luminance value at pixel point (i, j); v represents the luminance mean value of the neighborhood with (i, j) as the center and W×W as the size, and P(u, v) represents the probability of f(u, v) occurring; The exposure weight E is compared with the luminance value μ at the maximum two-dimensional entropy between the N images, and the reference luminance value μ is defined as: μ(i,j) = L nmax (i,j) Wherein, nmax represents the image sequence number n value corresponding to the maximum two-dimensional entropy at pixel point (i, j) between the N images, and the nmax value is obtained through the following formula: where H nmax represents the maximum two-dimensional entropy value at the pixel point (i, j), and the subscript nmax corresponds to the image sequence number corresponding to the maximum two-dimensional entropy at the pixel point (i, j) among the N images.

2. The adaptive-weight detail-preserving multi-exposure image fusion algorithm according to claim 1, wherein, In step three, With Solved from the equation: where ||·|| denotes the two-norm of a vector, contribute to the structural component of the fused image for each source image local block, max denotes the maximum value in the data group that follows, denotes the desired fused image block structure, s n denotes the unit length structure vector, denotes the mean-removed image block, n denotes the current image sequence index, and N denotes the number of source images to be fused.

3. The adaptive-weight detail-preserving multi-exposure image fusion algorithm according to claim 2, wherein, In step three, is represented as: The p value is determined by the structure consistency measure R between vectors, where ||·|| denotes the two-norm of a vector, n denotes the current image sequence index, N denotes the number of images to be fused, tan denotes the trigonometric tangent function, denotes the image block with mean removed.

4. The adaptive-weight detail-preserving multi-exposure image fusion algorithm according to claim 1, wherein, In step four, μ n The formula for calculating (i,j) is: μ n (i,j) = [I nR (i,j) + I nG (i,j) + I nB (i,j)] / 3.

5. The adaptive-weight detail-preserving multi-exposure image fusion algorithm according to claim 1, wherein, Step five is specifically: With the source image sequence I n as a reference, each corresponding weight map P n is guided filtered, and a guided weight map W n is output. W n = F r,ε (P n , I n ); where F r,ε denotes the guided filter, r, e are two constant parameters in the guided filter. After filtering the weight details, the N weight maps are normalized, and the normalized weight maps are added together are calculated from the following equation: where τ is a non-zero infinitesimal value, W n (i,j), respectively represent the guided filtered weight map and the normalized weight map.

Images