Rapid exposure fusion method based on multiple exposure

Abstract

The invention discloses a rapid exposure fusion method based on multiple exposures, and relates to the field of rapid exposure fusion, and the method comprises the steps: S1, inputting a plurality of LDR image sequences of different exposures; the method comprises the following steps: S101, estimating the overall illumination of each LDR image, and then sorting LDR image sequences from dark to bright to form new combined and sorted LDR images img1, img2 and img3; s102, on the basis of the LDR graph img2 in the middle of the sequence, calculating scene illumination T; s103, aiming at the scene illumination T, distributing the weight W of each LDR graph; s104, calculating a corresponding illuminance entropy k for each LDR image, and adjusting the illuminance of the corresponding image; s105, synthesizing an HDR image; and a multi-layer pyramid does not need to be constructed, so that the method has more advantages in speed and memory. Therefore, the algorithm is not affected even if the image lacks exposure time and ISO information. The algorithm is not limited by a small number of input images, and even only two images can be fused more naturally.

Description

Technical Field

[0001] The present invention relates to the technical field of rapid exposure fusion, and in particular to a rapid exposure fusion method based on multiple exposures. Background Art

[0002] Dynamic range is a common concept in physics, which refers to the ratio of the maximum value to the minimum value of a physical quantity, usually expressed in logarithms, with the unit of dB. For real scenes, it refers to the ratio of the illumination between the brightest and darkest parts of the scene. The illumination range of natural landscapes covers a very wide range, with a maximum dynamic range of up to 160dB, while the illumination range that the human eye can perceive is as high as 10 orders of magnitude. Since the range of natural light is from 120,000 Lux to 0.00035 Lux on a starry night, the indoor illumination is 100 Lux, while the illumination of the scenery outside may be 10,000 Lux. Therefore, when the camera looks out the window from indoors, the contrast between the two is 10,000 / 100 = 100:1. This contrast can be easily seen by the human eye because the human eye can handle a contrast ratio of 1000:1, but traditional cameras cannot because they generally only have a contrast ratio of 6:1. If higher contrast ratio is required, a more expensive camera is required. To solve this problem, there are many ways to synthesize HDR images by taking multiple LDR images with different exposures and using various algorithms.

[0003] At present, there are two main methods for synthesizing HDR images using multiple LDR images with different exposures. One is to construct a Gaussian pyramid and a Laplacian pyramid for multiple LDR images, retain the desired illumination, saturation, and contrast in each LDR, and fuse them into an HDR image. The other is to calculate the camera response curve from multiple LDR images, and then perform tone mapping on the multiple images to synthesize an HDR image.

[0004] For the first type of pyramid fusion, a multi-layer pyramid will be constructed for each LDR photo. When the number of LDRs and the number of pyramid layers increases, the computer memory will be greatly consumed and the calculation time will increase. In addition, the pyramid fusion method is difficult to integrate the effective pixels of a small area into HDR. For example, when shooting indoors out of a small window, even if the low-exposure image captures the outdoor scenery, it is difficult for the pyramid fusion to integrate it into the HDR.

[0005] For the method of calculating the camera response curve for tone mapping, when the number of LDR images is small, the fitted camera response curve will deviate greatly from the actual value. In addition, the calculation of the camera response curve requires the accurate exposure time and ISO value of each LDR image. When this information is missing, the camera response curve cannot be fitted. In addition, when there are many reflective objects in the scene, the tone mapping effect will be very unnatural.

[0006] To this end, we provide a fast exposure fusion method based on multiple exposures to solve the above problems. Summary of the invention

[0007] In view of the problems existing in the above-mentioned prior art, the present invention provides a fast exposure fusion method based on multiple exposures, which does not require the construction of a multi-layer pyramid, and thus has advantages in speed and memory. Therefore, even if the image lacks exposure time and ISO information, it has no effect on the algorithm. The algorithm is also not limited by the small number of input images, and even if there are only two images, the fusion can be relatively natural.

[0008] In order to achieve the above object, the present invention adopts a fast exposure fusion method based on multiple exposures, comprising:

[0009] S1: Input multiple LDR image sequences with different exposures;

[0010] S101: Estimate the overall illumination of each LDR image, and then sort the LDR image sequence from dark to bright to form a new combined sorted LDR image: img1, img2, img3;

[0011] S102: Calculate the scene illumination T based on the LDR image img2 in the middle of the sequence;

[0012] S103: Allocate a weight W to each LDR image according to the scene illumination T;

[0013] S104: Calculate the corresponding illumination entropy k for each LDR image, and adjust the illumination of the corresponding image;

[0014] S105: Synthesize an HDR image.

[0015] As a further optimization of the above scheme, when estimating the overall illumination of each LDR image, that is, taking the average of each channel of each pixel, and then taking the average of all pixels of the entire image, the illumination estimation constant value I of the image is obtained. n , where n represents the nth photo, and then the illumination of I1, I2, and I3 are sorted from dark to bright, and img1, img2, and img3 are sorted accordingly.

[0016] As a further optimization of the above solution, the pixel includes R / G / B channels.

[0017] As a further optimization of the above scheme, the scene illumination T is calculated. The middle image img2 is selected after sorting. First, it is normalized, and the pixel value is mapped from the original 0-255 to 0-1. Then, the maximum value of each channel of img2 is taken to obtain the initial illumination L(x), where x is a pixel.

[0018] L(x)=maxc∈{R,G,B} img2(x);

[0019] Where c is the channel, the maximum value in RGB is selected, img2(x): the pixel value of the second image;

[0020] The ideal illumination has local consistency for regions with similar structures. T retains the meaningful structure in the image and removes the texture edges. T is optimized by solving the following optimization equation:

[0021]

[0022] Among them, M is the weight matrix, and λ is a constant used to balance the proportion of the second smooth illumination. is to minimize the difference between the initial illuminance L and the scene illuminance T, Make the scene illumination T smoother;

[0023] Among them, M and Both have vertical and horizontal components;

[0024] The weight matrix M is:

[0025]

[0026] Among them, d represents different directions, vertical or horizontal, "|*|" is the absolute value operator, the lower part of the formula is a local window centered on pixel x, y belongs to a pixel in the local window, and v represents the vertical component. is the derivative of pixel y in the vertical or horizontal direction, It means that in the local window ω(x) centered at pixel x in the initial illumination L, y is differentiated in the vertical or horizontal direction, and then the sum is taken within the range of ω(x), where ∈ is a small arbitrary constant;

[0027] Will The equation is approximately:

[0028]

[0029] in, It is the derivative of the scene illumination T in the vertical or horizontal direction;

[0030] From the above formula, we can see that the only unknown quantity is T and the strongly related

[0031] In the process of finding T, quadratic terms are involved, and the solution is obtained directly by solving the following linear function:

[0032]

[0033] in, is element-wise division, I is the identity matrix, the operator Diag(v) constructs a diagonal matrix with the vector v, and Dd is the matrix obtained from the discrete gradient operator with forward differences.

[0034] As a further optimization of the above solution, illumination weights w1, w2, and w3 are assigned to each image, and appropriate parts of the images with different illuminations are considered for cropping, and weights with different ratios are designed.

[0035] As a further optimization of the above solution, the illumination entropies k1, k2, and k3 of each image are calculated, and the illuminations of each image are modified to obtain i1, i2, and i3; where, i n is the image after adjusting the illumination, img n is the input image; n represents the nth photo; a and b are camera constant parameters, and the camera performance suitable for the input image is adjusted; only the unknown k n needs to be calculated, which is used as an optimization function, and in the interval k n ∈{m, n}, where m and n are constants and m < n. The golden section search and parabolic interpolation can be used to find the local minimum k of the unary function n .

[0036] As a further optimization of the above solution, the HDR image is calculated. Each of the images i1, i2, and i3 after adjusting the illumination is multiplied element-wise with the corresponding weight, and then each result is accumulated and added to obtain the HDR result.

[0037] A fast exposure fusion method based on multiple exposures of the present invention has the following beneficial effects:

[0038] A fast exposure fusion method based on multiple exposures of the present invention does not require the construction of a multi-layer pyramid, so it has more advantages in terms of speed and memory. Therefore, even if the exposure time and ISO information of the image are missing, it has no impact on the algorithm. The algorithm is also not limited by the small number of input images. Even if there are only two images, they can be fused more naturally.

[0039] Referring to the following description and the accompanying drawings, specific embodiments of the present invention are disclosed in detail, indicating the ways in which the principles of the present invention can be adopted. It should be understood that the embodiments of the present invention are not limited in scope thereby. Within the spirit and terms of the appended claims, the embodiments of the present invention include many changes, modifications, and equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flowchart of the fast exposure fusion method based on multiple exposures of the present invention;

[0041] Figure 2It is a weight curve diagram of the present invention. DETAILED DESCRIPTION

[0042] Please refer to the instruction manual Figure 1-2 The present invention provides a technical solution: a fast exposure fusion method based on multiple exposures, comprising:

[0043] Input: A sequence of multiple LDR images at different exposures.

[0044] The following description takes three differently exposed images (Img1, Img2, Img3) as an example.

[0045] Step 1: Estimate the overall illumination of each LDR image, and then sort the LDR image sequence from dark to bright to form a new combined sorted LDR image (img1, img2, img3).

[0046] Step 2: Calculate the scene illumination T based on the LDR image img2 in the middle of the sequence.

[0047] Step 3: Assign a weight W to each LDR image for the scene illumination T.

[0048] Step 4: Calculate the corresponding illumination entropy k for each LDR image and adjust the illumination of the corresponding image.

[0049] Step 5: Synthesize the HDR image.

[0050] Detailed process description:

[0051] Step 1: Calculate the illumination estimation value for each image with different exposures, that is, take the average of each channel (R / G / B channel) of each pixel, and then take the average of all pixels in the entire image to obtain the illumination estimation constant value I of the image. n , where n represents the nth photo. Then sort the illumination of I1, I2, and I3 from dark to bright, and sort img1, img2, and img3 accordingly.

[0052] Step 2: Calculate the scene illumination T. Select the middle image img2 after sorting, and first normalize it, mapping the pixel value from the original 0-255 to 0-1. Then, take the maximum value of each channel (R / G / B channel) of img2 to get the initial illumination L(x), where x is a pixel.

[0053] L(x)=max c∈{R,G,B} img2(x);

[0054] Where c is the channel and the maximum value of RGB is selected. img2(x): The pixel value of the second image.

[0055] The ideal illumination should be locally consistent for regions with similar structures. T should preserve the meaningful structures in the image and remove texture edges. T is optimized by solving the following optimization equation:

[0056]

[0057] Among them, M is the weight matrix and λ is a constant used to balance the proportion of the second smooth illumination. is to minimize the difference between the initial illuminance L and the scene illuminance T, It makes the scene illumination T smoother.

[0058] Among them, M and There are vertical and horizontal components. The specific meaning and calculation are explained in detail below. The weight matrix M is:

[0059]

[0060] Where d represents different directions, vertical or horizontal, and “|*|” (* in the above formula represents Here we mainly explain the meaning of the "|*|" symbol) is the absolute value operator. The lower part of the formula is a local window centered on pixel x, y belongs to a pixel in the local window, d∈{h,v}h represents the horizontal component, v represents the vertical component, is the derivative of pixel y in the vertical or horizontal direction, It means that in the local window ω(x) centered at pixel x in the initial illumination L, y is differentiated vertically or horizontally, and then summed within the range of ω(x). ∈ is a very small constant (an arbitrary constant, not defined, just a minimum value to avoid the denominator being zero) to avoid zero denominator.

[0061] To reduce complexity, The equation is approximately:

[0062]

[0063] in, It is the derivative of the scene illumination T in the vertical or horizontal direction.

[0064] From the above formula, we can see that the only unknown quantity is T and the strongly related

[0065] In the process of finding T, only quadratic terms are now involved, and the solution is obtained directly by solving the following linear function:

[0066]

[0067] in, is element-wise division, I is the identity matrix, the operator Diag(v) constructs a diagonal matrix with the vector v, and Dd is the matrix obtained from the discrete gradient operator with forward differences.

[0068] Step 3: Assign the illumination weights w1, w2, and w3 for each image. Consider intercepting appropriate parts of images with different illuminations and designing weights with different ratios. Among them, the weight curves are as Figure 2 shown.

[0069]

[0070] w3 = (1 - T) 3 ;

[0071] Step 4: Calculate the illumination entropy k1, k2, and k3 for each image, and modify the illumination of each image to obtain i1, i2, and i3. The following formula is used to adjust the image illumination with the camera response curve, which is applicable to most cameras. Among them, i n is the image after adjusting the illumination, img n is the input image; n represents the nth photo; a and b are camera constant parameters, which can be adjusted to better suit the camera performance of the input image. Then, only the unknown k n needs to be calculated, which is used as an optimization function, and in the interval k n ∈{m, n}, where m and n are constants and m < n. The golden section search and parabolic interpolation can be used to find the local minimum k n .

[0072]

[0073] Step 5: Calculate the HDR image. Each adjusted illumination image i1, i2, and i3 is multiplied element-wise with the corresponding weight, and then each result is accumulated and added to obtain the HDR result.

[0074] HDR = ∑ n＝1,2,3 i n .*w n

[0075] Generate the HDR image by estimating the scene illumination. This method can be faster and less resource-consuming compared with other methods, and can also be better fused for relatively small areas.

[0076] The weight assignment method in Step 3. Retains the moderately exposed parts of different exposure images, avoids overexposure or underexposure, and the fusion is more natural.

[0077] There is no need to build a multi-layer pyramid, so it has advantages in speed and memory. The illumination of the input image is estimated, so even if the image lacks exposure time and ISO information, it has no effect on the algorithm. The algorithm is not limited by the small number of input images, and even if there are only two images, the fusion can be more natural.

[0078] It is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent substitution or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A fast exposure fusion method based on multiple exposures, characterized in that: include: S1: Input multiple LDR image sequences with different exposures; S101: Estimate the overall illumination of each LDR image, and then sort the LDR image sequence from dark to bright to form a new combined sorted LDR image: img1, img2, img3; S102: Calculate the scene illumination T based on the LDR image img2 in the middle of the sequence; S103: assigning a weight W to each LDR image according to the scene illumination T; S104: Calculate the corresponding illumination entropy k for each LDR image, and adjust the illumination of the corresponding image; S105: Synthesize an HDR image.

2. The method for rapid exposure fusion based on multiple exposures according to claim 1, characterized in that: in, When estimating the overall illumination of each LDR image, the illumination estimation constant value I of the image is obtained by taking the average of each channel of each pixel and then taking the average of all pixels of the entire image. n , where n represents the nth photo, and then the illumination of I1, I2, and I3 are sorted from dark to bright, and img1, img2, and img3 are sorted accordingly.

3. The method for rapid exposure fusion based on multiple exposures according to claim 2, characterized in that: A pixel includes R / G / B channels.

4. The method for rapid exposure fusion based on multiple exposures according to claim 1, characterized in that: Calculate the scene illumination T, select the middle image img2 after sorting, first normalize it, map the pixel value from the original 0-255 to 0-1, then take the maximum value of each channel of img2 to get the initial illumination L(x), where x is a pixel; L(x)=max c∈{R,G,B} img2(x); Where c is the channel, the maximum value in RGB is selected, img2(x): the pixel value of the second image; The ideal illumination has local consistency for regions with similar structures. T retains the meaningful structure in the image and removes the texture edge. T is optimized by solving the following optimization equation: Among them, M is the weight matrix, and λ is a constant used to balance the proportion of the second smooth illumination. is to minimize the difference between the initial illuminance L and the scene illuminance T, Make the scene illumination T smoother; Among them, M and Both have vertical and horizontal components; The weight matrix M is: Among them, d represents different directions, vertical or horizontal, "|*|" is the absolute value operator, the lower part of the formula is a local window centered on pixel x, y belongs to a pixel in the local window, and v represents the vertical component. is the derivative of pixel y in the vertical or horizontal direction, It means that in the local window ω(x) centered at pixel x in the initial illumination L, y is differentiated in the vertical or horizontal direction, and then the sum is taken within the range of ω(x), where ∈ is a small arbitrary constant; Will The equation is approximately: in, It is the derivative of the scene illumination T in the vertical or horizontal direction; From the above formula, we can see that the only unknown quantity is T and the strongly related In the process of finding T, quadratic terms are involved, and the solution is obtained directly by solving the following linear function: in, is element-by-element division, I is the identity matrix, the operator Diag(v) constructs a diagonal matrix using the vector v, and Dd is the matrix obtained by the discrete gradient operator with forward difference.

5. The method for rapid exposure fusion based on multiple exposures according to claim 1, characterized in that: Assign illumination weights w1, w2, w3 to each image, consider capturing appropriate parts of images with different illumination levels, and design weights with different proportions.

6. The method for rapid exposure fusion based on multiple exposures according to claim 1, characterized in that: Calculate the illumination entropy k1, k2, k3 of each picture, and modify the illumination of each picture to obtain i1, i2, i3; Among them, i n is the picture after adjusting the illumination, img n is the input picture; n represents the nth photo; a and b are camera constant parameters, which are adjusted to suit the camera performance of the input image; only the unknown k n needs to be calculated, which is used as an optimization function, and in the interval k n ∈{m, n}, where m and n are constants and m < n. The golden section search and parabolic interpolation can be used to find the local minimum k of the unary function n .

7. The method for rapid exposure fusion based on multiple exposures according to claim 1, characterized in that: Calculate the HDR image, adjust the illumination of each image i1, i2, i3 and the corresponding weight, multiply them element by element, and then add up each result to get the HDR result.

Images