A multi-exposure image fusion method

Abstract

A multi-exposure image fusion method belongs to the field of image processing. The method includes the following steps: S1: Calculate the average brightness of each input image and sort them in ascending order of average brightness; S2: Perform intensity mapping on the middle image so that its average brightness is at the middle gray value; for the remaining images, perform intensity mapping according to their ranking numbers so that the input image sequence contains images with various exposure levels; S3: Use the middle image as a reference image, divide each image into blocks, perform dynamic image detection and replacement block by block. S4: Perform multi-exposure fusion on the image sequence that has undergone dynamic detection and replacement to obtain the final fused image. The input image set of this method contains images with various exposure levels, improving the fusion effect; and the pre-dynamic image detection can screen out the moving objects in each image, replace them and perform the final fusion, improving the dynamic range of the image and avoiding the ghosting caused by moving objects.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a multi-exposure image fusion method. Background Art

[0002] Compared with the pictures taken by most digital cameras, natural scenes often have a higher dynamic range. This is because the response curve of traditional image sensors to light is linear, while the response of the human eye to light is more linear-logarithmic, so the human eye can capture more information. In order to present all the detailed information in the scene more realistically, a common method is to use different strategies to take multiple pictures of the same scene and use their complementarity to obtain a fused image that can richly reflect the scene information. Among them, the multi-exposure image fusion method is to fuse a series of images with different exposure times taken of the same scene. This method is relatively economical and effective, and an HDR image with rich information can be obtained using a linear image sensor with a low cost.

[0003] However, traditional multi-exposure image fusion algorithms require that the input image sequence be completely aligned, and slight shaking of the camera during shooting or the movement of objects in the scene will cause the appearance of ghosting. Therefore, most of the input image sequences of existing multi-exposure image fusion methods have to undergo strict image registration and do not allow the inclusion of moving objects.

[0004] Therefore, the present invention proposes a multi-exposure image fusion algorithm for enhancing the image quality and removing ghosting of an image sequence based on image preprocessing and structural similarity detection, so as to obtain a fused image of higher quality. Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-exposure image fusion method, which can achieve the fusion of images containing moving objects by improving the quality of the source images and removing ghosting. The fused image has more detailed information and can better restore the detailed information in the scene.

[0006] The present invention provides a multi-exposure image fusion method, comprising the following steps:

[0007] S1: Calculate the average brightness of each image, and sort the input image sequence in ascending order of exposure level according to the average brightness;

[0008] S2: Perform intensity mapping on each image to adjust the exposure level distribution of the image sequence sorted in step S1, thereby improving the quality of the source image sequence.

[0009] S3: Detect and replace dynamic objects in the image sequence processed in step S2, so that each image in the input image sequence no longer contains objects with inconsistent movements.

[0010] S4: Perform multi-exposure image fusion to obtain an HDR image.

[0011] Furthermore, the image quality enhancement based on S2 mainly includes the following steps:

[0012] S21: Normalize the pixel values of the image sequence and map all pixel values to the range [0, 1].

[0013] S22: Select the sorted and normalized middle image in step S1, and map the exposure level of the middle image to the 18% gray level according to its average brightness.

[0014] S23: For other images, determine the scaling factor according to their positions in the image sequence so that the image sequence contains images with various exposure levels.

[0015] S24: Renormalize the pixel values of each image, remap the pixel values that exceeded the value range due to scaling back to between 0 and 1, and then convert the image pixel values back to 8-bit integers.

[0016] Furthermore, the dynamic object detection and removal based on S3 includes the following steps:

[0017] S31: Select the middle image as the reference image.

[0018] S32: Sequentially select the images in the image sequence as the images to be detected, perform histogram matching on the reference image to make its histogram distribution similar to that of the image to be detected, and call the result of histogram matching on the reference image the mimetic image.

[0019] S33: Sharpen the mimetic image and the image to be detected, and only retain the structural contour information in the image, which is recorded as the structure diagram.

[0020] S34: For the two structure diagrams, sequentially take the N*N image blocks at the same position, and calculate the structural similarity factor of the two image blocks.

[0021] S35: Sequentially take the N*N image blocks at the same position of the mimetic image and the image to be detected, and calculate the brightness similarity factor and the contrast similarity factor of the two image blocks.

[0022] S36: Combine the three similarity factors to detect the positions of the image blocks in all the images to be detected whose structures do not match the mimetic image.

[0023] S37: For the image blocks that fail the similarity detection, subtract the image block at the same position in the sharpened image of the mimetic image from the image block at the same position in the sharpened image of the image to be detected, record the positions of the pixel points whose results are not 0, and perform the dilation operation in morphological processing on them to obtain the moving pixel point distribution map.

[0024] S38. For the motion pixel point distribution map obtained in the previous step, use the pixel points at the corresponding positions in the mimetic image to replace the motion pixel points in the image blocks that failed the similarity detection.

[0025] The further image fusion based on S4 includes the following steps:

[0026] S41. Normalize the image after detecting and replacing the dynamic objects.

[0027] S42. Calculate the luminance weight map of the image sequence based on luminance. The specific method is to first perform mean filtering on the image sequence, that is, for each pixel point, calculate the average value of the pixel points in the 3*3 matrix centered on it, and then obtain the luminance weight map according to the luminance values of the pixel points at the same positions in each image after mean filtering.

[0028] S43. Perform edge detection on each image and use the edge image as the contrast weight map.

[0029] S44. For each pixel point of each image, take the three-channel pixel values, calculate the average value of the absolute values of the differences between the three channels and the average value of the three channels; use their ratio as the color saturation weight map.

[0030] S45. Multiply the luminance weight map, contrast weight map, and color saturation weight map obtained in steps S42, S43, and S44 and perform normalization to obtain the final multi-exposure image fusion weight map.

[0031] S46. Obtain the Gaussian pyramid of the multi-exposure image fusion weight map sequence and the Laplacian pyramid of the image sequence.

[0032] S47. According to the Gaussian pyramid of the multi-exposure image fusion weight map, fuse the Laplacian pyramid of the image sequence to obtain the Laplacian pyramid of the fused image, and finally reconstruct the Laplacian pyramid of the fused image to obtain the fused image.

[0033] The further calculation of the luminance weight map in step S42 includes the following steps:

[0034] S421. Perform mean filtering on the image sequence, that is, for a pixel point in the image sequence, construct a new image sequence with the average value of the 3*3 window centered on it, and then calculate the average value of the pixel points at the same positions in the image sequence after mean filtering, denoted as the window average.

[0035] S422. Calculate the ideal median according to the window average.

[0036] S422. Calculate the luminance weight map according to the ideal median and the normal distribution function.

[0037] The further step S43 of calculating the contrast weight map includes the following steps:

[0038] S431. For any image U in the image sequence k , calculate its horizontal gradient as the horizontal gradient weight map;

[0039] S432. For any image U in the image sequence k , calculate its vertical gradient as the vertical gradient weight map.

[0040] S433. Merge the horizontal gradient weight map and the vertical gradient weight map to obtain the contrast weight map R of image U k . k .

[0041] This patent proposes a multi-exposure fusion algorithm based on structural similarity applicable to dynamic scenes. This algorithm can comprehensively extract detailed information from a set of low-dynamic images with different exposure levels taken of the same scene and combine them into an image with a high dynamic range. In addition, in actual shooting, due to camera shake or object displacement, the result of multi-exposure image fusion often produces unnecessary ghosting, affecting the imaging quality. However, the algorithm proposed in this patent can effectively eliminate the ghosting phenomenon in complex scenes and generate a ghost-free high-dynamic image, which can be used to improve the dynamic range of ordinary commercial cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 FIG. is a schematic diagram of the overall process of multi-exposure image fusion provided by an embodiment of the present invention.

[0043] Figure 2 FIG. is a schematic diagram of the process of regulating the exposure level of a multi-exposure image sequence provided by an embodiment of the present invention.

[0044] Figure 3 FIG. is a schematic diagram of the process of detecting and replacing moving objects provided by an embodiment of the present invention.

[0045] Figure 4 FIG. is the Laplacian pyramid fusion flow chart provided by an embodiment of the present invention.

[0046] Referring to Figure 1 , an embodiment of the present invention provides a multi-exposure fusion method, and the method includes the following steps:

[0047] S1. Calculate the average brightness of each image, and the formula is as follows:

[0048]

[0049] Then, sort the input image sequence in ascending order of exposure level according to the average brightness of the images. Among them, Ek represents the average brightness of the image, M and N represent the size of the image, and f k (x, y) represents the pixel value of the input image.

[0050] S2. Perform intensity mapping on each image, adjust the exposure level distribution of the image sequence sorted in step S1, enhance the exposure level distribution of the source image sequence, and thereby improve the quality of the source image sequence.

[0051] Step S2 can be detailed as the following operations:

[0052] S21. After normalizing the pixel values of all images, map all pixel values to the range [0, 1]. Take the image in the middle of the sequence and perform intensity mapping on it so that its average brightness is close to 18% gray level (which is 127 for 8-bit pixel values). The formula is as follows:

[0053]

[0054] where P r (x, y) represents the pixel value of the image after intensity mapping, and f r (x, y) represents the pixel value of the image without intensity mapping, and Er is the average brightness of the intermediate image obtained before normalization.

[0055] S22. For the remaining images, determine the scaling factor according to the sequence number of the intermediate image to make the image sequence contain various exposure levels. The specific calculation formula is as follows:

[0056]

[0057] where Q k (x, y) represents the pixel value of the image other than the intermediate image, and P k (x, y) represents the pixel value of the intermediate image after intensity mapping in S21, k is the sequence number of each image, and r is the sequence number of the intermediate image.

[0058] S23. Re-normalize the pixel values of all images to make the pixel values that exceed the image bit width limit due to scaling return to the normal range. The formula is

[0059]

[0060] where Q' k (x, y) represents the intensity mapping image after normalization, and Q k (x, y) represents the image after intensity mapping, and Q k_edge is the maximum pixel value in the k-th image. Through pixel value convergence, it is possible to avoid incorrect pixel values caused by being truncated due to exceeding the bit width limit when converting the gray value back to 8 bits.

[0061] S24. Restore the image format to an 8-bit wide image.

[0062] S3: Detect and replace dynamic objects in the image sequence processed in step S2 so that each image in the input image sequence no longer contains objects with inconsistent motions.

[0063] Step S3 can be detailed as the following steps:

[0064] S31. Select the middle image as the reference image and perform detection of dynamic objects and removal of ghosting on it.

[0065] S32. Calculate the structural similarity factor S k

[0066] Among them, step S32 is specifically divided into the following steps:

[0067] S321. According to the image to be measured I k Perform histogram matching on the reference image I r to obtain a mimetic image with a histogram distribution similar to that of the image to be measured. The formula is HM represents the histogram matching operation.

[0068] S322. Perform a sharpening operation on the mapped image and the image to be detected. The sharpening operation will highlight the contours of the objects and weaken the non-edge information to obtain the sharpened contour map C of the reference image r and the sharpened contour map C of the image to be measured k .

[0069] The operator used for sharpening is the Laplace operator

[0070] The specific formula is

[0071] C(x,y) = |I(x - 1,y) + I(x,y - 1) + I(x + 1,y) + I(x,y + 1) - 4I(x,y)|

[0072] where I(x,y) is the 8-bit image obtained by inverse normalization of Q'(x,y)

[0073] S323. For the sharpened maps C r and C k take N×N image blocks A at the same positions k and A r respectively, and perform structural similarity detection on each image block. The specific formula is (λ k is the structural similarity index of the image block A k and Ar)

[0074] where σ kr is the covariance of image block A k and A r , and the specific calculation formula is

[0075]

[0076] where μ A and μ r are the average values of the structural similarity indices of image block A k and Ar respectively.

[0077] σ k , σ r are the contrasts of the image blocks, and the calculation formula is

[0078]

[0079] S324. Given a threshold T k , for the image blocks with a structural similarity greater than the threshold, they are considered to be structurally similar, while those less than the threshold are considered not to meet the structural similarity. Finally, the structural similarity factor S k of each image block is obtained.

[0080]

[0081] S33. Calculate the luminance similarity factor l k and the contrast similarity factor w k

[0082] Step S33 can be specifically divided into the following steps:

[0083] S331. Perform histogram matching on the reference image I k according to the image I to be measured r to obtain a mimetic image with a histogram distribution similar to that of the image to be measured . The formula is where HM represents the histogram matching operation.

[0084] S332. Take image blocks D k of size N×N from the image I to be measured and the mapped image k and D r , and calculate the average luminance μ k , μ r and the contrast σ k , σ r of the two image blocks respectively.

[0085] S333. Calculate the image blocks D k , D rBrightness similarity index and contrast similarity index.

[0086]

[0087] where τ k is the brightness similarity index, and ε is a constant to avoid a denominator of 0.

[0088]

[0089] where is the contrast similarity index.

[0090] S**334**: Calculate the brightness similarity factor l k and the contrast similarity factor w k . If it is greater than the threshold, it is considered similar in terms of brightness and contrast; otherwise, it is considered that the brightness and contrast similarity is insufficient, and the similarity factor is set to 0.

[0091]

[0092]

[0093] S**34**: Obtain the final similarity factor based on the comprehensive consideration of the contrast similarity factor, brightness similarity factor, and structural similarity factor, and calculate the similarity factor B k :

[0094] B k = S k × (l k + w k )

[0095] If B k = 0, it is considered that the image block to be measured contains a moving object.

[0096] S**35**: Select the moving object and perform replacement.

[0097] The further step S**35** can be detailed as the following steps:

[0098] S**351**: Perform a sharpening operation on the mimetic image of the reference image to obtain the sharpened mimetic image s

[0099] S**352**: Subtract the sharpened image C k and and take the absolute value to obtain the contour difference map [[ID=..]]

[0100] S**353**: For the image block containing the moving object, we take the image block V d at the corresponding position in the contour difference map Cd , after obtaining the non-zero elements among them, perform the dilation operation in morphological processing, that is, for the image block V d all elements within a certain radius r around the non-zero elements in it are regarded as moving pixel points, obtaining a distribution map of moving pixel points, and then replace the moving pixel points in the image to be measured with the pixel points at the corresponding positions in the mimic image.

[0101] S4: Perform multi-exposure image fusion to obtain an HDR image

[0102] Further, S4 can be detailed as the following steps:

[0103] S41. Normalize the image after detecting and replacing dynamic objects

[0104] S42. Calculate the brightness weight map G based on the image exposure level according to the exposure levels of each pixel point in the image sequence k

[0105] Further, S42 specifically includes the following steps:

[0106] S421. For any pixel point P k (x, y), for the 3*3 matrix centered on it, calculate its average value θ k , obtaining the image sequence after mean filtering;

[0107] S422. Calculate the window average value of the pixel points at the same position in the image sequence after mean filtering, and the calculation formula is

[0108]

[0109] where L is the number of image sources.

[0110] S423. Calculate the window theoretical median γ, γ = 0.25 + 0.5θ, and the pixel points with a small difference between the pixel value and the theoretical median will have a higher brightness weight.

[0111] S424. Calculate the weight distribution at the point (x, y) of the image sequence, and the formula is

[0112]

[0113] where σ is a constant used to control the distribution amplitude of the normal distribution function in terms of brightness; calculate the brightness weight map according to the ideal median and the normal distribution function.

[0114] S43. Calculate the contrast weight map based on the image contrast according to the results of edge detection in the image sequence.

[0115] Further, S43 specifically includes the following steps:

[0116] S431. For any image U in the image sequence k , first use the sobel operator in the horizontal direction to calculate its horizontal gradient, which is the horizontal gradient weight map A k . The sobel operator in the horizontal direction is The calculation formula is

[0117] A k (x, y) = |U k (x + 1, y - 1) + 2U k (x + 1, y) + U k (x + 1, y + 1) - U k (x - 1, y - 1) - 2U k (x - 1, y) - U k (x - 1, y + 1)|

[0118] S432. Then use the sobel operator in the vertical direction to calculate its vertical gradient, which is the vertical gradient weight map B k . The sobel operator in the vertical direction is The calculation formula is

[0119] B k (x, y) = |U k (x - 1, y - 1) + 2U k (x, y - 1) + U k (x + 1, y - 1) - U k (x - 1, y + 1) - 2U k (x, y + 1) - U k (x + 1, y + 1)|

[0120] S433. Combine the horizontal gradient and the vertical gradient to obtain the contrast weight map R k of the image U k . The specific formula is

[0121] S44. Calculate the color weight map according to the color information of each point in the image sequence.

[0122] Step S44 can be detailed as the following steps:

[0123] S441. For any pixel point P k (x, y), take the pixel values of its three channels and calculate the average value of the absolute differences between the three channels The calculation formula is

[0124]

[0125] S442. Calculate the average value of the pixel values of the three channels The calculation formula is

[0126]

[0127] S443. Calculate the ratio between the two, which is the color weight Y corresponding to this pixel point k , and the calculation formula is

[0128]

[0129] S45. Multiply the three weights and perform normalization to obtain the final weight distribution map ω k . Note that at this time, the weight map in the image sequence needs to be normalized so that for any pixel, the sum of the pixel weights at this position in the image sequence is 1

[0130] S46. Perform Gaussian pyramid operation on the weight map and Laplacian pyramid operation on the input image, and each layer of the image pyramid is fused according to the weight pyramid

[0131] Step S46 can be detailed as the following steps

[0132] S461. Perform Gaussian pyramid operation on both the weight map sequence and the image sequence

[0133] The specific operation of the Gaussian pyramid in step S461 is as follows

[0134] S4611. Perform n-layer Gaussian pyramid on an image A k , first take A k as the first layer of the pyramid

[0135] S4612. Downsample the image to obtain the downsampled image The formula is where 1:2:end means taking pixel points every other row and every other column of the image matrix, and finally obtaining an image with half the length and width

[0136] S4613. Perform Gaussian filtering on the downsampled image , and the filtering operator is The second-layer Gaussian image can be obtained by using Gaussian filtering The specific calculation formula is

[0137]

[0138] S4614. After continuously operating n - 1 times, a Gaussian pyramid of an image can be obtained

[0139] ​S462. Derive the Laplacian pyramid sequence from the Gaussian pyramid of the source image sequence.

[0140] The Laplacian pyramid operation in step S462 can be detailed as the following steps:

[0141] S4621. Take the nth layer of the Gaussian pyramid of the image as the nth layer of the Laplacian pyramid.

[0142] S4622. Upsample the image to obtain an upsampled image. The specific operation is to insert 0s between each row and each column of to finally form an upsampled image with the same size as the next layer of the pyramid.

[0143] S4623. Apply Gaussian filtering to the upsampled image to obtain an image Subtract it from The resulting residual image is the (n - 1)th layer of the Laplacian pyramid, and the calculation formula is

[0144]

[0145] S463. Fuse the Laplacian pyramid images of the same layer of the source image sequence according to the corresponding layer Gaussian pyramid of the weight map sequence to obtain the Laplacian pyramid of the fused image {i = 1:n|F i}. The calculation formula is

[0146]

[0147] S464. Reconstruct the Laplacian pyramid of the fused image to obtain the fused image: Upsample the top layer image F n to obtain an upsampled image, then apply Gaussian filtering to obtain a filtered image and add it to the next layer; adding layer by layer in this way can obtain the final fused image F.​

Claims

1. A multi-exposure image fusion method, characterized in that It includes the following steps: S1: Calculate the average brightness of each image, and sort the input image sequence in ascending order of exposure level according to the average brightness; S2: Perform intensity mapping on each image to adjust the exposure level distribution of the image sequence sorted in step S1, thereby improving the quality of the source image sequence; S3: Detect and replace dynamic objects in the image sequence processed in step S2, so that each image in the input image sequence no longer contains objects with inconsistent motion; S4: Perform multi-exposure image fusion to obtain an HDR image; The step S3 includes the following steps: S31. Select the middle image as the reference image; S32. Select the images in the image sequence as the images to be detected in turn, perform histogram matching on the reference image to make its histogram distribution similar to that of the image to be detected, and call the result of histogram matching on the reference image the mimetic image; S33. Sharpen the mimetic image and the image to be detected, and only retain the structural contour information in the image, which is recorded as the structure diagram; S34. For the two structure diagrams, take the N*N image blocks at the same position in turn, and calculate the structural similarity factor of the two image blocks; S35. Take the N*N image blocks at the same position of the mimetic image and the image to be detected in turn, and calculate the brightness similarity factor and contrast similarity factor of the two image blocks; S36. Combine the three similarity factors to detect the positions of the image blocks in all the images to be detected whose structures do not match the mimetic image; S37. For the image blocks that fail the similarity detection, subtract the image blocks at the same position in the sharpened image of the mimetic image from the image blocks at the same position in the sharpened image of the image to be detected, record the positions of the pixel points whose results are not 0, and perform the dilation operation in morphological processing to obtain the moving pixel point distribution map; S38. For the moving pixel point distribution map recorded in the previous step, use the pixel points at the corresponding positions in the mimetic image to replace the moving pixel points in the image blocks that fail the similarity detection.

2. The multi-exposure image fusion method according to claim 1, wherein, The step S2 includes the following steps: S21. Normalize the pixel values of the image sequence, and map all pixel values to the range of [0,1]; S22. Take the middle image sorted and normalized in step S1, and map the exposure level of the middle image to the 18% gray level according to its average brightness; S23. For other images, determine the scaling factor according to their positions in the image sequence, so that the image sequence contains images with various exposure levels; S24. Renormalize the pixel values of each image, remap the pixel values that exceed the value range due to scaling back to between 0 and 1, and then convert the image pixel values back to 8-bit integers.

3. A multi-exposure image fusion method according to claim 1, characterized in that The image fusion based on S4 includes the following steps: S41. Normalize the image after detecting and replacing dynamic objects; S42. Obtain the brightness weight map of the image sequence based on brightness. The specific method is to first perform mean filtering on the image sequence, that is, for each pixel point, calculate the average value of the pixel points in the 3*3 matrix centered on it, and then obtain the brightness weight map according to the brightness values of the pixel points at the same position in each mean-filtered image; S43. Perform edge detection on each image, and use the edge image as the contrast weight map; S44. For each pixel of each image, take the pixel values of the three channels, calculate the average value of the absolute differences of the three channels and the average value of the three channels; use their ratio to create a color saturation weight map. S45. Multiply the brightness weight map, contrast weight map, and color saturation weight map obtained in steps S42, S43, and S44 and perform normalization to obtain the final multi-exposure image fusion weight map. S46. Obtain the Gaussian pyramid of the multi-exposure image fusion weight map sequence and the Laplacian pyramid of the image sequence. S47. According to the Gaussian pyramid of the multi-exposure image fusion weight map, fuse the Laplacian pyramid of the image sequence to obtain the Laplacian pyramid of the fused image, and finally reconstruct the Laplacian pyramid of the fused image to obtain the fused image.

4. A multi-exposure image fusion method according to claim 3, wherein The calculation of the brightness weight map in step S42 includes the following steps: S421. For each pixel in the image sequence, replace its pixel value with the average value of the 3×3 window centered on it, and then calculate the average value of the pixel values of the pixels at the same position in the image sequence after mean filtering, which is denoted as the window mean. S422. Calculate the ideal median according to the window mean. S422. Calculate the brightness weight map according to the ideal median and the normal distribution function.

5. A multi-exposure image fusion method according to claim 3, characterized in that The calculation of the contrast weight map in step S43 includes the following steps: S431. For any image U in the image sequence k , calculate its horizontal gradient as the horizontal gradient weight map; S432. For any image U in the image sequence k , calculate its vertical gradient as the vertical gradient weight map; S433. Merge the horizontal gradient weight map and the vertical gradient weight map to obtain the contrast weight map R of image U k k .​

Images