Multi-exposure image fusion method and system based on local detail reservation

Abstract

The invention relates to the technical field of multi-exposure image fusion, in particular to a multi-exposure image fusion method and system based on local detail preservation, and the method comprises the following steps: obtaining an exposure image sequence; calculating a multi-scale local weight of each exposure image in the exposure image sequence; calculating the detail weight of each exposure image in the exposure image sequence; calculating the significance weight of each exposure image in the exposure image sequence; calculating a fusion weight map based on the multi-scale local weight, the detail weight and the significance weight; and performing double-pyramid decomposition and fusion on the exposure image sequence and the fusion weight map to obtain a fusion image. According to the method, the local information is captured by adopting the multi-scale blocks, so that the global feature capturing capability of the larger blocks is reserved, the accurate description of the local texture change by the smaller blocks is ensured, richer feature representation can be provided for subsequent processing tasks such as image fusion, and the image fusion efficiency is improved. Therefore, the performance in the aspects of detail reservation, color information and the like is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-exposure image fusion, and in particular, to a multi-exposure image fusion method and system based on local detail retention. Background Art

[0002] With the rapid development of digital imaging technology, people's requirements for image quality are increasing day by day. Due to the limitation of the dynamic range of the sensor, low dynamic range images are difficult to capture both the highlight areas and dark details in the scene at the same time, resulting in the images being unable to truly reflect the rich information of the real scene, and it has been difficult to meet the requirements for high-quality images in modern applications.

[0003] High dynamic range (HDR) images can show a wider brightness range and richer detail information than traditional images, and have a wide range of applications in fields such as photography and computer vision. Multi-exposure image fusion (MEF), as an effective way to generate HDR images simply and efficiently, has received extensive attention in recent years. MEF effectively expands the dynamic range of the image and retains more detail information by fusing multiple images taken with different exposure parameters.

[0004] Although existing MEF methods can achieve precise fusion in static scenes, in dynamic scenes, their ghosting removal performance varies significantly. In recent years, scholars at home and abroad have carried out research on ghosting removal in dynamic scenes. Oguzhan Ulucan et al. proposed ghost-free multi-exposure image fusion applicable to static and dynamic scenes. The weight chart characterization process depends on the principal component weight, the adaptive exposure factor weight, and the saliency weight. This method can effectively improve the ghosting removal performance in dynamic scenes, but in terms of details, its detail retention performance is difficult to be satisfactory. In addition, color deviation will occur in the images fused by this method.

[0005] In order to effectively enhance image details and solve the color deviation phenomenon, the present invention provides a new multi-exposure image fusion method with local detail retention. Summary of the Invention

[0006] The purpose of the present invention is to provide a multi-exposure image fusion method and system based on local detail retention, which captures local information by multi-scale partitioning. It not only retains the global feature capture ability of larger partitions but also ensures the accurate description of local texture changes by smaller partitions, and can provide richer feature representations for subsequent image fusion and other processing tasks, so as to improve the performance in aspects such as detail retention and color information, and solve the technical problems pointed out in the background art.

[0007] The present invention is realized through the following technical solutions: A multi-exposure image fusion method based on local detail retention, comprising the following steps: Obtain an exposure image sequence; Calculate the multi-scale local weights of each exposure image in the exposure image sequence; Calculate the detail weights of each exposure image in the exposure image sequence; Calculate the saliency weights of each exposure image in the exposure image sequence; Calculate a fusion weight map based on the multi-scale local weights, detail weights, and saliency weights; Perform bi-pyramid decomposition and fusion on the exposure image sequence and the fusion weight map to obtain a fused image.

[0008] According to a preferred embodiment, before calculating the weights of the exposure image sequence, the method further includes converting the multi-exposure image sequence from an RGB image to a grayscale image.

[0009] According to a preferred embodiment, calculating the multi-scale local weights of each exposure image in the exposure image sequence specifically includes: Perform multi-scale partitioning on each exposure image in the exposure image sequence to obtain image blocks of different scales; Independently calculate feature parameters within the image blocks of different scales to obtain the local information of each exposure image at different scales; Fuse the local information of each exposure image at different scales, and calculate the weight map of each type of local information; Perform information retention processing on the weight maps of various local information according to a preset information retention strategy; Merge the weight maps of each type of local information after information retention processing to construct a local weight map.

[0010] According to a preferred embodiment, independently calculating feature parameters within the image blocks of different scales specifically includes: calculating the information entropy and contrast of each image block.

[0011] According to a preferred embodiment, fusing the local information of each exposure image at different scales specifically includes: fusing the local information of each exposure image at different scales through the L2 norm to obtain an information entropy weight map and a contrast weight map.

[0012] According to a preferred embodiment, the preset information retention strategy is: if the local feature parameter is a larger value among all pixels, assign a weight of 1 to this pixel point, otherwise assign a weight of 0.

[0013] According to a preferred embodiment, calculating the detail weights of each exposure image in the exposure image sequence specifically includes: Perform edge enhancement on the exposure images with different exposure degrees; Use the switching contrast operator to extract the image edge details of the exposure image after the edge enhancement process, obtaining the image details generated by dilation and the image details generated by erosion; Take the absolute value difference between the image details generated by dilation and the image details generated by erosion to generate a detail weight map.

[0014] According to a preferred embodiment, calculate the saliency weights of each exposure image in the exposure image sequence, specifically including: Perform a discrete cosine transform on the input exposure image to obtain the DCT coefficient matrix in the frequency domain; Take the sign of the DCT coefficient matrix; Calculate the inverse DCT of the sign matrix and reconstruct it into an image; Normalize the reconstructed image to obtain a saliency weight map.

[0015] According to a preferred embodiment, calculate the fusion weight map based on the multi-scale local weights, detail weights, and saliency weights, specifically including: Merge the local weight map, detail weight map, and saliency weight map; Normalize the merged weight map; Use guided filtering to filter the normalized weight map to obtain a fusion weight map.

[0016] The present invention also provides a multi-exposure image fusion system based on local detail preservation, which is applied to the multi-exposure image method as described above. The system includes: An image acquisition module for acquiring an exposure image sequence; A first processing module for calculating the multi-scale local weights of each exposure image in the exposure image sequence; A second processing module for calculating the detail weights of each exposure image in the exposure image sequence; A third processing module for calculating the saliency weights of each exposure image in the exposure image sequence; An image combination module for calculating a fusion weight map based on the multi-scale local weights, detail weights, and saliency weights; A pyramid reconstruction module for performing dual pyramid decomposition and fusion on the exposure image sequence and the fusion weight map to obtain a fused image.

[0017] The technical solution of a multi-exposure image fusion method and system based on local detail retention provided by the present invention has at least the following advantages and beneficial effects: (1) By using multi-scale partitioning to capture local information, the present invention not only retains the global feature capture ability of larger partitions but also ensures the accurate description of local texture changes by smaller partitions, providing a richer feature representation for subsequent processing tasks such as image fusion, thereby improving performance in aspects such as detail retention and color information; (2) The adopted preset information retention strategy can ensure that important detail areas are not lost during subsequent processing, thus retaining the details of the source image sequence in the fused image and further improving performance in aspects such as detail retention and color information; (3) The improvement in aspects such as detail retention and color information is not only reflected in a single scene but also shows high stability and consistency in different scenes, being able to adapt to diverse practical application requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic flowchart of the multi-exposure image fusion method provided in Embodiment 1 of the present invention; Figure 2 It is a schematic principle diagram of the multi-exposure image fusion method provided in Embodiment 1 of the present invention; Figure 3 It is a schematic principle diagram of the preset information retention strategy provided in Embodiment 1 of the present invention; Figures 4 to 6 It is a schematic diagram of the results of the comparative experiment provided in Embodiment 1 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Usually, the components of the embodiments of the present invention described and illustrated herein can be arranged and designed in various different configurations.

[0020] Embodiment 1 To improve the performance of the multi-exposure image fusion method in aspects such as detail retention and color information, the present invention proposes a multi-exposure image fusion method based on local detail retention.

[0021] The multi-exposure image fusion method based on local detail retention of the present invention captures local information by using multi-scale partitioning, which not only retains the global feature capture ability of larger partitions but also ensures the accurate description of local texture changes by smaller partitions, providing a richer feature representation for subsequent processing tasks such as image fusion, thereby improving performance in aspects such as detail retention and color information.

[0022] The multi-exposure image fusion method based on local detail preservation in this embodiment is as follows Figure 1 and Figure 2 shown, and includes the following steps: Step 1: Obtain a sequence of exposure images; In some embodiments, in the same high-dynamic range scene, a same device is used to take pictures at different exposure times to obtain multiple images with different exposure degrees, forming a sequence of exposure images; further, after converting the sequence of exposure images from RGB images to grayscale images, the sequence of exposure images to be fused is obtained.

[0023] Step 2: Calculate the multi-scale local weights of each exposure image in the sequence of exposure images to be fused, so as to retain the local information for capturing the image; In some embodiments, calculating the multi-scale local weights of each exposure image in the sequence of exposure images to be fused specifically includes: Step 2.1: Perform multi-scale partitioning on each exposure image in the sequence of exposure images to obtain image blocks of different scales. The partitioning rules for the multi-scale partitioning are as follows:

[0024]

[0025] In the above formula, represents the size of the first-level partition, is the height of the exposure image, is the width of the exposure image, represents the level partition size, represents the level partition size.

[0026] Step 2.2: Independently calculate feature parameters within image blocks of different scales to obtain local information of each exposure image at different scales.

[0027] In some embodiments, independently calculating feature parameters within image blocks of different scales in the same exposure image specifically includes: calculating the information entropy and contrast of each image block. The information entropy is used to measure the randomness and complexity of the grayscale distribution of the exposure image, and the contrast captures local structural features through the gray-level co-occurrence matrix.

[0028] In this embodiment, the expression for calculating the information entropy is as follows:

[0029] In the above formula, represents the information entropy, represents the maximum gray level, represents the gray level, Indicates the probability of occurrence of the th gray level.

[0030] The expression for calculating the contrast is as follows:

[0031] In the above formula, represents the contrast, represents the gray level, represents the pixel point in the gray-level co-occurrence matrix probability value.

[0032] Step 2.3: Fuse the local information of each of the exposure images at different scales, construct a weight map for each type of local information, so as to balance the relationship between global features and local details and avoid the problem of possible feature loss at a single scale.

[0033] In some embodiments, fusing the local information of each of the exposure images at different scales specifically includes: fusing the local information of each of the exposure images at different scales through the L2 norm to obtain an information entropy weight map and a contrast weight map, where the expression for fusing the local information corresponding to the information entropy is as follows:

[0034] In the above formula, represents the information entropy weight at the pixel point , represents the th scale of information entropy; it should be noted that after calculating the information entropy of the image block, it will be assigned to each position in the image block. For example, if the information entropy calculated for a 3*3 image block is 4, then a 3*3 matrix will be obtained, where each value is 4.

[0035] The expression for fusing the local information corresponding to the contrast is as follows:

[0036] In the above formula, represents the contrast weight at the pixel point , represents the th scale of contrast.

[0037] Step 2.4: Perform information retention processing on the weight maps of the various local information according to a preset information retention strategy.

[0038] In some embodiments, refer to Figure 3As shown in the figure, the preset information retention strategy is as follows: If the characteristic parameter, such as the local information entropy of this pixel, is a relatively large value among all pixels, it indicates that this pixel corresponds to an important detail part in the exposure image, and a higher weight needs to be assigned to it to ensure that this detail part will not be lost during subsequent processing; specifically, in this embodiment, a weight of 1 is assigned to this pixel, otherwise a weight of 0 is assigned.

[0039] In this embodiment, the expression for performing information retention processing on the information entropy weight map is as follows:

[0040]

[0041] In the above formula, represents the information entropy weight map after information retention processing, represents the th image at pixel The information entropy weight at the position, represents the set of characteristic parameters ranked before among all pixels of the corresponding image, represents the number of input image sequences.

[0042] The expression for performing information retention processing on the contrast weight map is as follows:

[0043] In the above formula, represents the contrast weight map after information retention processing, represents the th image at pixel The contrast weight at the position.

[0044] Specifically, the preset information retention strategy adopted in the embodiments of the present invention can ensure that important detail areas will not be lost during subsequent processing, so as to retain the details of the source image sequence in the fused image, and further improve the performance in aspects such as detail retention and color information.

[0045] Step 2.5: Merge the weight maps of each local information after information retention processing to construct a local weight map.

[0046] In some embodiments, the expression for merging the weight maps of each local information after information retention processing is as follows:

[0047] In the above formula, represents the local weight map.

[0048] Step Three: Calculate the detail weights of each exposure image in the exposure image sequence; In some embodiments, calculating the detail weights of each exposure image in the exposure image sequence specifically includes: Step 3.1: Before extracting details, perform edge enhancement on the exposure images with different exposure degrees.

[0049] Step 3.2: Use the switching contrast operator TOC to extract the image edge details of the exposure image after the edge enhancement process, obtaining the image details generated by dilation and the image details generated by erosion. Among them, the definition of the switching contrast operator TOC is as follows:

[0050] In the above formula, represents the gray value at pixel point after being processed by the switching contrast operator, represents the dilation operation, represents the erosion operation, represents the original exposure image at pixel point with the gray value of and

[0051] represents the structural element. Since the gray value of the dilation result is not less than the gray value of the original exposure image, in this embodiment, the calculation expression of the image details generated by dilation is as follows:

[0052] In the above formula, represents the gray value of the image details generated by dilation at pixel point represents the original image after applying operation at pixel point with the gray value of

[0053] Correspondingly, the calculation expression of the image details generated by erosion is as follows:

[0054] In the above formula, represents the gray value of the image details generated by erosion at pixel point

[0055] Step 3.3: Take the absolute value difference between the image details generated by dilation and the image details generated by erosion to generate the detail weight, and the expression is as follows:

[0056] In the above formula, represents the detail weight, Represents the grayscale value of the image details generated by dilation. Represents the grayscale value of the image details generated by erosion.

[0057] Step Four: Calculate the saliency weights of each exposure image in the exposure image sequence. In some embodiments, calculating the saliency weights of each exposure image in the exposure image sequence specifically includes: Step 4.1: Perform a discrete cosine transform on the input exposure image to transform the exposure image from the spatial domain to the frequency domain, obtaining a DCT coefficient matrix in the frequency domain to prominently represent the energy of different frequency components. The expression is as follows:

[0058] In the above formula, Represents the DCT coefficient matrix. , Both represent normalization factors. , Respectively represent the number of pixels of the input exposure image in the horizontal and vertical directions. Represents the pixel At the position in the spatial domain of the input exposure image. , Respectively represent the coordinate variables in the frequency domain, used to index different frequency components. Corresponds to the horizontal frequency. Corresponds to the vertical frequency.

[0059] Step 4.2: Take the sign of the DCT coefficient matrix, that is, calculate the positive and negative relationship of each DCT coefficient matrix. The expression is as follows:

[0060] In the above formula, Represents the processing result of taking the sign of the DCT coefficient matrix .

[0061] Step 4.3: Calculate the inverse DCT of the sign matrix and reconstruct it into an image. The expression is as follows:

[0062] In the above formula, Represents the reconstructed image.

[0063] Step 4.4: Perform normalization processing on the reconstructed image, map the pixel values of the reconstructed image to the range of 0 - 1, and obtain the saliency weight map.

[0064] Step Five: Calculate the fusion weight map based on the multi-scale local weights, detail weights, and saliency weights. In some embodiments, a fused weight map is calculated based on the multi-scale local weights, detail weights, and saliency weights, specifically including: Step 5.1: Combine the detail weight map, the detail weight map, and the saliency weight map. The expression is as follows:

[0065] In the above formula, represents the combined weight map, represents the detail weight map, represents the saliency weight map, represents the local weight map, is a very small positive value to ensure that the denominator is non-zero.

[0066] Step 5.2: Normalize the combined weight map

[0067] Step 5.3: Use guided filtering to filter the normalized weight map to obtain the final weight map. The expression is as follows:

[0068] In the above formula, represents the final weight map, represents the guided filtering operation.

[0069] Step Six: Perform double pyramid decomposition and fusion on the exposure image sequence and the fused weight map to obtain a fused image; In some embodiments, performing double pyramid decomposition and fusion on the exposure image sequence and the fused weight map specifically includes: Step 6.1: Calculate the fused image according to the following formula:

[0070] In the above formula, is the representation of the -th layer of the Laplacian pyramid decomposition of the fused image, is the representation of the -th layer of the Laplacian pyramid decomposition of the exposure image, is the representation of the -th layer of the Gaussian pyramid decomposition of the fused weight map.

[0071] Step 6.2: Starting from the highest layer, gradually upsample and stack the detail information of the fused Laplacian pyramid to reconstruct the final fused image. The expression is as follows:

[0072] In the above formula, ​Indicates the fused image after reconstruction at the layer of the pyramid. The corresponding to the 0th layer is the final fused image. represents the upsampling operation.

[0073] The following is an explanation of the comparative experiment on the multi-exposure image fusion method provided in this embodiment: Refer to Figures 4 to 6 as shown. Using MEF-SSIM, , Three objective evaluation indicators are used to conduct a comparative experiment on the images of current mainstream methods in 18 different scenarios. These indicators evaluate the image quality from different dimensions. The abscissa represents various existing image fusion methods, where Our is this method. MEF-SSIM mainly measures the structural similarity of the image and can effectively reflect the degree of preservation of the structural information of the image. The larger the MEF-SSIM, the more similar the structure of the fused image is to the original image. focuses on evaluating the ability to preserve edge details of the image and can reflect the edge enhancement effect of the algorithm. The larger it is, the more details are retained in the fused image. evaluates the contrast performance of the image from the perspective of visual effect and is consistent with the human eye visual perception characteristics.

[0074] From Figures 4 to 6 it can be seen that the present invention performs best among many methods, and the performance in the MEF-SSIM, , indicators is the highest, indicating that it has the best ability in preserving image structure, details, color information, etc., fully demonstrating its comprehensive performance advantages in the multi-exposure image fusion task. This advantage is not only reflected in a single scenario, but also shows high stability and consistency in different scenarios, and can adapt to diverse actual application requirements.

[0075] Embodiment 2 Based on the technical solution provided in Embodiment 1, this embodiment provides a multi-exposure image fusion system based on local detail preservation. This system applies the multi-exposure image method provided in Embodiment 1. The system includes: An image acquisition module for acquiring an exposure image sequence; a first processing module for calculating multi-scale local weights of each exposure image in the exposure image sequence; a second processing module for calculating detail weights of each exposure image in the exposure image sequence; a third processing module for calculating saliency weights of each exposure image in the exposure image sequence; an image combination module for calculating a fusion weight map based on the multi-scale local weights, detail weights, and saliency weights; a pyramid reconstruction module for performing dual pyramid decomposition and fusion on the exposure image sequence and the fusion weight map to obtain a fused image.

[0076] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A multi-exposure image fusion method based on local detail preservation, characterized in that: The steps include: Get exposure image sequence; Calculating a multi-scale local weight of each exposure image in the exposure image sequence; Calculating the detail weight of each exposure image in the exposure image sequence; Calculating the saliency weight of each exposure image in the exposure image sequence; Calculating a fusion weight map based on the multi-scale local weights, detail weights, and saliency weights; The exposure image sequence and the fusion weight map are subjected to dual pyramid decomposition and fusion to obtain a fused image.

2. The multi-exposure image fusion method based on local detail preservation according to claim 1, characterized in that: Before performing weight calculation on the exposure image sequence, the method further includes converting the multiple exposure image sequence from an RGB image to a grayscale image.

3. The multi-exposure image fusion method based on local detail preservation as claimed in claim 2, characterized in that: Calculating the multi-scale local weight of each exposure image in the exposure image sequence specifically includes: Perform multi-scale block division on each exposure image in the exposure image sequence to obtain image blocks of different scales; Independently calculating characteristic parameters in the image blocks of different scales to obtain local information of different scales of each of the exposure images; Fusing the local information of different scales of each of the exposure images, and calculating a weight map of each type of local information; Performing information retention processing on the weight graphs of the various local information according to a preset information retention strategy; The weight maps of each type of local information after information preservation processing are merged to construct a local weight map.

4. The multi-exposure image fusion method based on local detail preservation according to claim 3, characterized in that: The characteristic parameters are calculated independently in the image blocks of different scales, specifically including: calculating the information entropy and contrast of each image block.

5. The multi-exposure image fusion method based on local detail preservation as claimed in claim 3, characterized in that: The local information of different scales of each of the exposure images is fused, specifically comprising: fusing the local information of different scales of each of the exposure images through the L2 norm to obtain an information entropy weight map and a contrast weight map.

6. The multi-exposure image fusion method based on local detail preservation according to claim 4, characterized in that: The preset information retention strategy is: if the local feature parameter is a larger value among all pixels, a weight of 1 is assigned to the pixel point, otherwise a weight of 0 is assigned.

7. The multi-exposure image fusion method based on local detail preservation according to any one of claims 1 to 6, characterized in that: Calculating the detail weight of each exposure image in the exposure image sequence specifically includes: Perform edge enhancement on exposure images with different exposure levels; Using a switching contrast operator to extract image edge details of the exposed image after the edge enhancement process, to obtain image details generated by dilation and image details generated by erosion; The image details generated by dilation and the image details generated by erosion are subtracted in absolute value to generate a detail weight map.

8. The multi-exposure image fusion method based on local detail preservation according to any one of claims 1 to 6, characterized in that: Calculating the saliency weight of each exposure image in the exposure image sequence specifically includes: Perform discrete cosine transform on the input exposure image to obtain the DCT coefficient matrix in the frequency domain; Taking a sign for the DCT coefficient matrix; Calculate the inverse DCT of the symbol matrix and reconstruct it into an image; The reconstructed image is normalized to obtain a saliency weight map.

9. The multi-exposure image fusion method based on local detail preservation according to any one of claims 1 to 6, characterized in that: Calculating a fusion weight map based on the multi-scale local weights, detail weights, and saliency weights specifically includes: Merge the local weight map, detail weight map and saliency weight map; Normalize the merged weight map; The normalized weight map is filtered using guided filtering to obtain a fused weight map.

10. A multi-exposure image fusion system based on local detail preservation, characterized in that: Applied to the multi-exposure image method according to any one of claims 1 to 9, the system comprises: An image acquisition module, used for acquiring an exposure image sequence; A first processing module, used for calculating the multi-scale local weight of each exposure image in the exposure image sequence; A second processing module, used for calculating the detail weight of each exposure image in the exposure image sequence; A third processing module, used for calculating the significance weight of each exposure image in the exposure image sequence; An image combination module, used for calculating a fusion weight map based on the multi-scale local weights, detail weights and significance weights; The pyramid reconstruction module is used to perform double pyramid decomposition and fusion on the exposure image sequence and the fusion weight map to obtain a fused image.

Images