A multi-exposure image fusion method and system based on local detail preservation

Abstract

The present invention relates to the technical field of multi-exposure image fusion, and more specifically to a multi-exposure image fusion method and system based on local detail preservation, comprising the following steps: obtaining an exposure image sequence; calculating multi-scale local weights for each exposure image in the exposure image sequence; calculating detail weights for each exposure image in the exposure image sequence; calculating saliency weights for each exposure image in the exposure image sequence; calculating a fusion weight map based on the multi-scale local weights, detail weights, and saliency weights; and performing dual-pyramid decomposition and fusion on the exposure image sequence and the fusion weight map to obtain a fused image. By employing multi-scale blocking to capture local information, the present invention not only retains the ability of larger blocks to capture global features, but also ensures that smaller blocks accurately describe local texture changes. This method can provide richer feature representations for subsequent processing tasks such as image fusion, thereby improving performance in terms of detail preservation and color information.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-exposure image fusion, and in particular to a multi-exposure image fusion method and system based on local detail preservation. Background Art

[0002] With the rapid development of digital imaging technology, people's requirements for image quality are increasing. Due to the limitations of the sensor's dynamic range, low dynamic range images struggle to capture both bright and dark details in a scene. As a result, the images fail to truly reflect the rich information of real scenes, making it difficult to meet the high-quality image requirements of modern applications.

[0003] High dynamic range (HDR) images can display a wider brightness range and richer detail than traditional images, and have broad applications in photography, computer vision, and other fields. Multi-exposure image fusion (MEF) has garnered significant attention in recent years as a simple and efficient method for generating HDR images. MEF effectively expands the image's dynamic range and preserves more detail by fusing multiple images captured with different exposure parameters.

[0004] Although existing MEF methods can achieve accurate fusion in static scenes, their deghosting performance varies significantly in dynamic scenes. In recent years, domestic and foreign scholars have conducted research on deghosting in dynamic scenes. Oguzhan Ulucan et al. proposed a ghosting-free multi-exposure image fusion method suitable for both static and dynamic scenes. The weight map representation process relies on the principal component weights, adaptive exposure factor weights, and saliency weights. This method can effectively improve the deghosting performance of dynamic scenes, but its detail preservation performance is less than satisfactory. In addition, the images fused using this method will also show color cast.

[0005] In order to effectively enhance image details and solve the color cast phenomenon, the present invention provides a new multi-exposure image fusion method with local detail preservation. Summary of the Invention

[0006] The purpose of the present invention is to provide a multi-exposure image fusion method and system based on local detail preservation, which uses multi-scale blocking to capture local information, retaining the ability of larger blocks to capture global features while ensuring the accurate description of local texture changes by smaller blocks. It can provide richer feature representation for subsequent processing tasks such as image fusion, thereby improving performance in detail preservation and color information, and solving the technical problems pointed out in the background technology.

[0007] The present invention is implemented through the following technical solution: a multi-exposure image fusion method based on local detail preservation, comprising the following steps:

[0008] Get exposure image sequence;

[0009] Calculating a multi-scale local weight of each exposure image in the exposure image sequence;

[0010] Calculating a detail weight of each exposure image in the exposure image sequence;

[0011] Calculating the saliency weight of each exposure image in the exposure image sequence;

[0012] Calculating a fusion weight map based on the multi-scale local weights, detail weights, and saliency weights;

[0013] The exposure image sequence and the fusion weight map are subjected to dual pyramid decomposition and fusion to obtain a fused image.

[0014] According to a preferred embodiment, before performing weight calculation on the exposure image sequence, the method further comprises converting the multi-exposure image sequence from an RGB image to a grayscale image.

[0015] According to a preferred embodiment, calculating the multi-scale local weight of each exposure image in the exposure image sequence specifically includes:

[0016] Performing multi-scale block division on each exposure image in the exposure image sequence to obtain image blocks of different scales;

[0017] Independently calculating feature parameters in the image blocks of different scales to obtain local information of different scales of each of the exposure images;

[0018] fusing local information of different scales of each of the exposure images and calculating a weight map for each type of local information;

[0019] Performing information retention processing on the weight graphs of the various local information according to a preset information retention strategy;

[0020] The weight maps of each local information after information preservation processing are merged to construct a local weight map.

[0021] According to a preferred embodiment, independently calculating characteristic parameters in the image blocks of different scales specifically includes calculating information entropy and contrast of each image block.

[0022] According to a preferred embodiment, the local information of different scales of each of the exposure images is fused, specifically comprising: fusing the local information of different scales of each of the exposure images through the L2 norm to obtain an information entropy weight map and a contrast weight map.

[0023] According to a preferred embodiment, the preset information retention strategy is: if the local feature parameter is a larger value among all pixels, a weight of 1 is assigned to the pixel point, otherwise a weight of 0 is assigned.

[0024] According to a preferred embodiment, calculating the detail weight of each exposure image in the exposure image sequence specifically includes:

[0025] Perform edge enhancement on exposure images with different exposure levels;

[0026] extracting image edge details of the exposed image after the edge enhancement process using a switching contrast operator, and obtaining image details generated by dilation and image details generated by erosion;

[0027] The absolute value of the image details generated by dilation and the image details generated by erosion are subtracted to generate a detail weight map.

[0028] According to a preferred embodiment, calculating the saliency weight of each exposure image in the exposure image sequence specifically includes:

[0029] Perform discrete cosine transform on the input exposure image to obtain the DCT coefficient matrix in the frequency domain;

[0030] Taking the sign of the DCT coefficient matrix;

[0031] Calculate the inverse DCT of the symbol matrix and reconstruct it into an image;

[0032] The reconstructed image is normalized to obtain the saliency weight map.

[0033] According to a preferred embodiment, calculating a fusion weight map based on the multi-scale local weights, detail weights, and saliency weights specifically includes:

[0034] Merge the local weight map, detail weight map and saliency weight map;

[0035] Normalize the merged weight map;

[0036] The normalized weight map is filtered using guided filtering to obtain a fused weight map.

[0037] The present invention further provides a multi-exposure image fusion system based on local detail preservation, which is applied to the multi-exposure image method described above, and includes:

[0038] An image acquisition module, used for acquiring an exposure image sequence;

[0039] A first processing module, configured to calculate a multi-scale local weight of each exposure image in the exposure image sequence;

[0040] a second processing module, configured to calculate a detail weight of each exposure image in the exposure image sequence;

[0041] a third processing module, configured to calculate a saliency weight of each exposure image in the exposure image sequence;

[0042] An image combination module, configured to calculate a fusion weight map based on the multi-scale local weights, detail weights, and saliency weights;

[0043] The pyramid reconstruction module is used to perform dual-pyramid decomposition and fusion on the exposure image sequence and the fusion weight map to obtain a fused image.

[0044] The technical solution of a multi-exposure image fusion method and system based on local detail preservation provided by the present invention has at least the following advantages and beneficial effects: (1) The present invention uses multi-scale blocking to capture local information, which not only retains the ability of larger blocks to capture global features, but also ensures the accurate description of local texture changes by smaller blocks, and can provide richer feature representations for subsequent processing tasks such as image fusion, thereby improving performance in detail preservation and color information; (2) The preset information retention strategy adopted can ensure that important detail areas will not be lost in the subsequent processing process, thereby retaining the details of the source image sequence in the fused image, further improving performance in detail preservation and color information; (3) The improvement in detail preservation and color information is not only reflected in a single scene, but also shows higher stability and consistency in different scenes, and can adapt to diverse practical application needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of the process of the multi-exposure image fusion method provided in Example 1 of the present invention;

[0046] Figure 2 A schematic diagram illustrating the principle of the multi-exposure image fusion method provided in Example 1 of the present invention;

[0047] Figure 3 A schematic diagram illustrating the principle of the preset information retention policy provided in Example 1 of the present invention;

[0048] Figures 4 to 6 This is a schematic diagram of the results of the comparative experiment provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0050] Example 1

[0051] In order to improve the performance of the multi-exposure image fusion method in terms of detail preservation and color information, the present invention proposes a multi-exposure image fusion method based on local detail preservation.

[0052] The multi-exposure image fusion method based on local detail preservation of the present invention uses multi-scale blocking to capture local information, which not only retains the ability of larger blocks to capture global features, but also ensures the accurate description of local texture changes by smaller blocks. It can provide richer feature representation for subsequent processing tasks such as image fusion, thereby improving performance in terms of detail preservation and color information.

[0053] The multi-exposure image fusion method based on local detail preservation in this embodiment is as follows: Figure 1 and Figure 2 As shown, the following steps are included:

[0054] Step 1: Obtain exposure image sequence;

[0055] In some embodiments, in the same high dynamic range scene, the same device is used to shoot at different exposure times to obtain multiple images with different exposure levels to form an exposure image sequence; further, the exposure image sequence is converted from RGB image to grayscale image to obtain the exposure image sequence to be fused.

[0056] Step 2: calculating the multi-scale local weight of each exposure image in the exposure image sequence to be fused, so as to retain the local information used to capture the image;

[0057] In some embodiments, calculating the multi-scale local weight of each exposure image in the exposure image sequence to be fused specifically includes:

[0058] Step 2.1: Perform multi-scale blocking on each exposure image in the exposure image sequence to obtain image blocks of different scales. The blocking rules of the multi-scale blocking are as follows:

[0059]

[0060]

[0061] In the above formula, Indicates the size of the first level block, is the height of the exposed image, is the width of the exposed image, Indicates the The size of the level block, Indicates the The size of the level block.

[0062] Step 2.2: independently calculating feature parameters in image blocks of different scales to obtain local information of different scales of each of the exposure images.

[0063] In some embodiments, feature parameters are independently calculated within image blocks of different scales in the same exposure image, specifically including: calculating the information entropy and contrast of each image block, where the information entropy is used to measure the randomness and complexity of the grayscale distribution of the exposure image, and the contrast is used to capture local structural features through the grayscale co-occurrence matrix.

[0064] In this embodiment, the expression for calculating information entropy is as follows:

[0065]

[0066] In the above formula, represents information entropy, Indicates the maximum gray level, Indicates the grayscale level, Indicates the The probability of a gray level appearing.

[0067] The expression for calculating contrast is as follows:

[0068]

[0069] In the above formula, Indicates contrast, Indicates the grayscale level, Represents the pixel point in the gray level co-occurrence matrix The probability value of .

[0070] Step 2.3: Fuse the local information of different scales of each exposure image and construct a weight map for each local information to balance the relationship between global features and local details and avoid the problem of feature loss that may exist at a single scale.

[0071] In some embodiments, fusing the local information of different scales of each of the exposure images specifically includes fusing the local information of different scales of each of the exposure images using the L2 norm to obtain an information entropy weight map and a contrast weight map, wherein the expression of the local information corresponding to the fused information entropy is as follows:

[0072]

[0073] In the above formula, Represents pixel points The information entropy weight at Indicates the The information entropy at each scale is calculated. It should be noted that after the information entropy of an image block is calculated, it will be assigned to each position in the image block. For example, if the information entropy of a 3*3 image block is calculated to be 4, then a 3*3 matrix will be obtained, in which each value is 4.

[0074] The expression for fusing the local information corresponding to the contrast is as follows:

[0075]

[0076] In the above formula, Represents pixel points The contrast weight at Indicates the Contrast ratio at each scale.

[0077] Step 2.4: Perform information retention processing on the weight graphs of the various local information according to a preset information retention strategy.

[0078] In some embodiments, see Figure 3 As shown, the preset information retention strategy is: if the characteristic parameter, such as the local information entropy of the pixel is a larger value among all pixels, it indicates that the pixel corresponds to an important detail part in the exposure image, and it needs to be given a higher weight to ensure that the detail part will not be lost in the subsequent processing process; specifically, this embodiment assigns a weight of 1 to the pixel, otherwise it assigns a weight of 0.

[0079] In this embodiment, the expression for performing information preservation processing on the information entropy weight map is as follows:

[0080]

[0081]

[0082] In the above formula, Represents the information entropy weight graph after information retention processing, Indicates the Image in pixels The information entropy weight at Indicates the ranking of all pixels in the corresponding image The previous feature parameter set, Indicates the number of input image sequences.

[0083] The expression for information preservation processing of contrast weight map is as follows:

[0084]

[0085] In the above formula, represents the contrast weight map after information preservation processing, Indicates the Image in pixels The contrast weight.

[0086] Specifically, the preset information retention strategy adopted in the embodiment of the present invention can ensure that important detail areas are not lost during subsequent processing, thereby retaining the details of the source image sequence in the fused image, further improving the performance in detail retention and color information.

[0087] Step 2.5: Merge the weight maps of each type of local information after information preservation processing to construct a local weight map.

[0088] In some embodiments, the expression for merging the weight maps of each type of local information after information preservation processing is as follows:

[0089]

[0090] In the above formula, represents the local weight map.

[0091] Step 3: Calculate the detail weight of each exposure image in the exposure image sequence;

[0092] In some embodiments, calculating the detail weight of each exposure image in the exposure image sequence specifically includes:

[0093] Step 3.1: Before extracting details, perform edge enhancement on the exposure images with different exposure levels.

[0094] Step 3.2: Use a switching contrast operator TOC to extract image edge details of the exposed image after the edge enhancement process, and obtain image details generated by dilation and image details generated by erosion. The switching contrast operator TOC is defined as follows:

[0095]

[0096] In the above formula, Indicates the pixel point after switching contrast operator processing The gray value at represents the expansion operation, represents the corrosion operation, Represents the original exposure image At the pixel The gray value at Represents a structural element.

[0097] Since the grayscale value of the dilation result will not be less than the grayscale value of the original exposure image, in this embodiment, the calculation expression of the image details generated by the dilation is as follows:

[0098]

[0099] In the above formula, Indicates the image details generated by dilation at the pixel point The gray value at Represents the original image application After operation at pixel point The gray value at .

[0100] Correspondingly, the calculation expression of the image details generated by corrosion is as follows:

[0101]

[0102] In the above formula, Indicates that the image details produced by corrosion are at the pixel point The gray value at .

[0103] Step 3.3: Subtract the absolute value of the image details generated by dilation from the image details generated by erosion to generate detail weights, which are expressed as follows:

[0104]

[0105] In the above formula, represents the detail weight, represents the grayscale value of the image details produced by dilation, Grayscale values ​​representing image details produced by erosion.

[0106] Step 4: Calculate the significance weight of each exposure image in the exposure image sequence;

[0107] In some embodiments, calculating the saliency weight of each exposure image in the exposure image sequence specifically includes:

[0108] Step 4.1: Perform discrete cosine transform on the input exposure image to convert the exposure image from the spatial domain to the frequency domain. The DCT coefficient matrix in the frequency domain is obtained to highlight the energy of different frequency components. The expression is as follows:

[0109]

[0110] In the above formula, represents the DCT coefficient matrix, 、 Both represent normalization factors, 、 Respectively represent the number of pixels in the horizontal and vertical directions of the input exposure image, Represents the pixels of the input exposure image in the spatial domain The gray value at 、 Respectively represent the coordinate variables in the frequency domain, used to index different frequency components, Corresponding horizontal frequency, Corresponding vertical frequency.

[0111] Step 4.2: Take the sign of the DCT coefficient matrix, that is, calculate the positive and negative relationship of each DCT coefficient matrix, as shown in the following expression:

[0112]

[0113] In the above formula, Represents the DCT coefficient matrix Get the result of symbol processing.

[0114] Step 4.3: Calculate the inverse DCT of the symbol matrix and reconstruct it into an image. The expression is as follows:

[0115]

[0116] In the above formula, Represents the reconstructed image.

[0117] Step 4.4: Normalize the reconstructed image and map the pixel values ​​of the reconstructed image to the range of 0-1 to obtain a saliency weight map.

[0118] Step 5: Calculate a fusion weight map based on the multi-scale local weights, detail weights, and saliency weights;

[0119] In some embodiments, calculating a fusion weight map based on the multi-scale local weights, detail weights, and saliency weights specifically includes:

[0120] Step 5.1: Combine the detail weight map, detail weight map, and saliency weight map. The expression is as follows:

[0121]

[0122] In the above formula, represents the combined weight graph, represents the detail weight map, represents the saliency weight map, represents the local weight map, A minimal positive value, ensuring that the denominator is non-zero.

[0123] Step 5.2: Merged weight graph Perform normalization processing.

[0124] Step 5.3: Use guided filtering to normalize the weight map Perform filtering to obtain the final weight map, which is expressed as follows:

[0125]

[0126] In the above formula, represents the final weight graph, Represents the guided filtering operation.

[0127] Step 6: performing dual pyramid decomposition and fusion on the exposure image sequence and the fusion weight map to obtain a fused image;

[0128] In some embodiments, performing dual-pyramid decomposition and fusion on the exposure image sequence and the fusion weight map specifically includes:

[0129] Step 6.1. Calculate the fused image according to the following formula:

[0130]

[0131] In the above formula, is the Laplacian pyramid decomposition of the fused image. Layer representation, is the first Laplacian pyramid decomposition of the exposure image. Layer representation, The first step of Gaussian pyramid decomposition of the fused weight graph Layer representation.

[0132] Step 6.2: Starting from the highest layer, gradually upsample and superimpose the details of the fused Laplacian pyramid to reconstruct the final fused image. The expression is as follows:

[0133]

[0134] In the above formula, Indicates that in the pyramid The fused image after layer reconstruction, the corresponding layer 0 This is the final fused image. Represents an upsampling operation.

[0135] The following is a comparative experiment on the multi-exposure image fusion method provided in this embodiment:

[0136] See also Figures 4 to 6 As shown, using MEF-SSIM, 、 Three objective evaluation indicators were used to conduct comparative experiments on images of 18 different scenes using the current mainstream methods. These indicators evaluate the image quality from different dimensions. The horizontal axes represent various existing image fusion methods, among which Our represents our method. MEF-SSIM mainly measures the structural similarity of images and can effectively reflect the degree of preservation of the structural information of the image. The larger the MEF-SSIM, the more similar the fused image structure is to the original image. It focuses on evaluating the image's ability to preserve edge details and can reflect the edge enhancement effect of the algorithm. The larger the value, the more details the fused image retains; The contrast performance of the image is evaluated from the perspective of visual effects, which is consistent with the visual perception characteristics of the human eye. The smaller it is, the more the fused image conforms to the human eye's perception habits.

[0137] Depend on Figures 4 to 6 It can be seen that the present invention performs best among many methods, MEF-SSIM, 、 The performance in all indicators is the highest, indicating that it has the best ability in preserving image structure, details, color information, etc., fully demonstrating its comprehensive performance advantage in multi-exposure image fusion tasks. This advantage is not only reflected in a single scene, but also shows high stability and consistency in different scenes, and can adapt to diverse practical application needs.

[0138] Example 2

[0139] This embodiment, based on the technical solution provided in Example 1, provides a multi-exposure image fusion system based on local detail preservation. The system is applied to the multi-exposure image method provided in Example 1. The system includes:

[0140] An image acquisition module is used to acquire an exposure image sequence; a first processing module is used to calculate the multi-scale local weights of each exposure image in the exposure image sequence; a second processing module is used to calculate the detail weights of each exposure image in the exposure image sequence; a third processing module is used to calculate the saliency weights of each exposure image in the exposure image sequence; an image combination module is used to calculate a fusion weight map based on the multi-scale local weights, detail weights, and saliency weights; and a pyramid reconstruction module is used to perform dual-pyramid decomposition and fusion on the exposure image sequence and the fusion weight map to obtain a fused image.

[0141] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A multi-exposure image fusion method based on local detail preservation, characterized in that: The steps include: Get exposure image sequence; Calculating the multi-scale local weight of each exposure image in the exposure image sequence, specifically comprising: performing multi-scale blocking on each exposure image in the exposure image sequence to obtain image blocks of different scales, independently calculating information entropy and contrast within the image blocks of different scales, fusing the information entropy and contrast of the different scales of each exposure image by L2 norm to obtain an information entropy weight map and a contrast weight map, performing information retention processing on the information entropy weight map and the contrast weight map according to a preset information retention strategy, merging the information entropy weight map and the contrast weight map after the information retention processing to construct a local weight map; Calculating detail weights of each exposure image in the exposure image sequence, specifically comprising: performing edge enhancement on exposure images of different exposure levels, extracting image edge details of the exposure images after the edge enhancement using a switching contrast operator, obtaining image details generated by dilation and image details generated by erosion, and performing an absolute value difference between the image details generated by dilation and the image details generated by erosion to generate a detail weight map; Calculating the saliency weight of each exposure image in the exposure image sequence, specifically comprising: performing a discrete cosine transform on the input exposure image to obtain a DCT coefficient matrix in the frequency domain, taking a sign of the DCT coefficient matrix, calculating an inverse DCT of the sign matrix, reconstructing the DCT coefficient matrix into an image, and normalizing the reconstructed image to obtain a saliency weight map; Calculating a fusion weight map based on the multi-scale local weights, detail weights, and saliency weights; The exposure image sequence and the fusion weight map are subjected to dual pyramid decomposition and fusion to obtain a fused image.

2. The multi-exposure image fusion method based on local detail preservation according to claim 1, characterized in that: Before performing weight calculation on the exposure image sequence, the method further includes converting the exposure image sequence from an RGB image to a grayscale image.

3. The multi-exposure image fusion method based on local detail preservation according to claim 2, characterized in that: The preset information retention strategy is: if the local information entropy and contrast are ranked among all pixels If the value is larger than the previous one, the pixel is assigned a weight of 1, otherwise it is assigned a weight of 0.

4. The multi-exposure image fusion method based on local detail preservation according to any one of claims 1 to 3, characterized in that: Calculating a fusion weight map based on the multi-scale local weights, detail weights, and saliency weights specifically includes: Merge the local weight map, detail weight map and saliency weight map; Normalize the merged weight map; The normalized weight map is filtered using guided filtering to obtain a fused weight map.

5. A multi-exposure image fusion system based on local detail preservation, characterized in that: Applied to the multi-exposure image fusion method based on local detail preservation according to any one of claims 1 to 3, the system includes: An image acquisition module, used for acquiring an exposure image sequence; A first processing module, configured to calculate a multi-scale local weight of each exposure image in the exposure image sequence; a second processing module, configured to calculate a detail weight of each exposure image in the exposure image sequence; a third processing module, configured to calculate a saliency weight of each exposure image in the exposure image sequence; An image combination module, configured to calculate a fusion weight map based on the multi-scale local weights, detail weights, and saliency weights; The pyramid reconstruction module is used to perform dual-pyramid decomposition and fusion on the exposure image sequence and the fusion weight map to obtain a fused image.

Images