Multi-exposure image fusion method and device, electronic equipment and computer readable storage medium

Abstract

This invention proposes a multi-exposure image fusion method, apparatus, electronic device, and computer-readable storage medium, relating to the field of image processing technology. The method generates a motion region weight map, a median threshold bitmap, a superpixel region map, and a base weight map to distinguish and model moving objects from static backgrounds in dynamic scenes. The motion region weight map is corrected using the median threshold bitmap and superpixel region map to improve the spatial consistency and boundary fit of motion detection. Motion interference in the base weight map is removed using the final motion weight map, resulting in an accurate fused weight map. Based on this, multi-exposure fusion is performed using the original image sequence, effectively suppressing ghosting artifacts caused by object motion while preserving highlight and shadow details, thus improving the quality and realism of the fused image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a multi-exposure image fusion method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Multi-exposure image fusion technology aims to generate a high-quality synthetic image with a wider dynamic range and richer details by integrating a sequence of images of the same scene acquired under different exposure conditions. However, traditional fusion measurements are prone to introducing visual artifacts such as ghosting in situations with drastic brightness changes or rapid object movement. Therefore, there is an urgent need for a multi-exposure image fusion method that can reduce computational complexity and effectively minimize visual artifacts, thereby improving the adaptability of multi-exposure image fusion under conditions of complex lighting and dynamic content. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a multi-exposure image fusion method, apparatus, electronic device and computer-readable storage medium that can effectively suppress ghosting artifacts caused by object motion and improve the quality and realism of the fused image.

[0004] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:

[0005] In a first aspect, the present invention provides a multi-exposure image fusion method, the method comprising:

[0006] Based on the original image sequence, a motion region weight map, a median threshold bitmap, a superpixel region map, and a basic weight map are generated for each original image. The motion region weight map represents the location region where dynamic objects and static backgrounds are located in the original image. The median threshold bitmap represents the binarized motion region obtained by median filtering of the original image. The superpixel region map represents the local region division in the original image whose boundary conforms to the object contour. The basic weight map represents the illumination quality and contribution weight of each pixel in the original image.

[0007] The motion region weight map is corrected based on all the median threshold bitmaps and the superpixel region map of each original image to obtain the final motion weight map corresponding to each original image.

[0008] Based on the final motion weight map corresponding to each original image, the motion interference of the corresponding basic weight map is removed to obtain the fusion weight map corresponding to each original image;

[0009] The fused image is obtained by fusing all the fusion weight maps and the original image sequence.

[0010] In an optional implementation, the step of generating a motion region weight map, a median threshold bitmap, a superpixel region map, and a basic weight map corresponding to each original image based on the original image sequence includes:

[0011] Select the best-exposed original image from the original image sequence as the reference image;

[0012] A motion region weight map corresponding to each original image is generated based on the grayscale difference, Euclidean distance, and structural similarity between each original image and the reference image.

[0013] Calculate the median grayscale value of each original image, and generate a median threshold bitmap corresponding to each original image based on the median grayscale value and the grayscale value of each pixel in the original image;

[0014] Cluster each pixel in each of the original images to generate a superpixel region map corresponding to each of the original images;

[0015] Static scene fusion is performed on each original image to obtain the basic weight map corresponding to each original image.

[0016] In an optional implementation, generating a motion region weight map corresponding to each original image based on the grayscale difference, Euclidean distance, and structural similarity between each original image and a reference image includes:

[0017] For each of the original images, a corresponding reference image block is determined in the reference image based on the coordinates of each pixel in the original image and a preset block size;

[0018] The pixel with the smallest grayscale value difference between the reference image block and the corresponding pixel in the original image is determined as the target pixel;

[0019] The grayscale difference of each pixel in the original image is determined based on the grayscale value of the target pixel and the grayscale value of the corresponding pixel in the original image;

[0020] The Euclidean distance for each pixel in the original image is determined based on the pixel coordinates of the target pixel and the corresponding pixel coordinates in the original image.

[0021] The structural similarity of each pixel in the original image is determined based on the image patch corresponding to each pixel in the original image and the reference image patch;

[0022] The motion weights corresponding to each pixel in the original image are determined based on the grayscale difference, Euclidean distance, and structural similarity of each pixel in the original image; the motion weights corresponding to all pixels in the original image constitute the motion region weight map corresponding to the original image.

[0023] In an optional implementation, the step of correcting the corresponding motion region weight map based on all the median threshold bitmaps and the superpixel region map of each original image to obtain the final motion weight map corresponding to each original image includes:

[0024] Generate a global median bitmap based on all the aforementioned median threshold bitmaps;

[0025] Based on the global median bitmap and the motion region weight map corresponding to each original image, an intermediate motion weight map corresponding to each original image is generated;

[0026] Based on the superpixel region map and the corresponding intermediate motion weight map of each original image, a final motion weight map is generated for each original image.

[0027] In an optional implementation, generating an intermediate motion weight map corresponding to each original image based on the global median bitmap and the motion region weight map corresponding to each original image includes:

[0028] The global median bitmap is processed using opening and closing operations to obtain the processed global median bitmap;

[0029] A temporary motion weight map corresponding to each original image is obtained by performing a pixel-by-pixel multiplication based on the motion region weight map corresponding to each original image and the processed global median bitmap.

[0030] Dilatation is performed on the temporary motion weight map corresponding to each original image to obtain the intermediate motion weight map corresponding to each original image.

[0031] In an optional implementation, the step of dilating the temporary motion weight map corresponding to each of the original images to obtain an intermediate motion weight map corresponding to each of the original images includes:

[0032] The total number of pixels and the number of moving pixels corresponding to each original image are determined based on the temporary motion weight map corresponding to each original image;

[0033] The number of dilation steps for each original image is determined based on the total number of pixels and the number of moving pixels corresponding to each original image;

[0034] By performing dilation processing on the temporary motion weight map corresponding to the dilation number of each original image, an intermediate motion weight map corresponding to each original image is obtained.

[0035] In an optional implementation, the step of fusing the entire fusion weight map and the original image sequence to obtain a fused image includes:

[0036] Construct a Gaussian pyramid based on the fusion weight graphs described above;

[0037] Construct the Laplacian pyramid based on the original image sequence;

[0038] The Gaussian pyramid and the Laplace pyramid are merged layer by layer, and the bottom layer image is determined as the merged image.

[0039] In a second aspect, the present invention provides a multi-exposure image fusion apparatus, the apparatus comprising:

[0040] The processing module is used to generate a motion region weight map, a median threshold bitmap, a superpixel region map, and a basic weight map for each original image based on the original image sequence. The motion region weight map represents the location regions of dynamic objects and static backgrounds in the original image. The median threshold bitmap represents the binarized motion region obtained by median filtering of the original image. The superpixel region map represents the local region division in the original image whose boundaries conform to the object contours. The basic weight map represents the illumination quality and contribution weight of each pixel in the original image.

[0041] The correction module is used to correct the corresponding motion region weight map based on all the median threshold bitmaps and the superpixel region map of each original image, so as to obtain the final motion weight map corresponding to each original image.

[0042] The fusion module is used to remove motion interference from the corresponding base weight map based on the final motion weight map corresponding to each original image to obtain a fused weight map corresponding to each original image; and to fuse all the fused weight maps and the original image sequence to obtain a fused image.

[0043] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing a computer program executable by the processor, the processor executing the computer program to implement the multi-exposure image fusion method described in any of the foregoing embodiments.

[0044] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-exposure image fusion method as described in any of the foregoing embodiments.

[0045] Compared to existing technologies, the multi-exposure image fusion method, apparatus, electronic device, and computer-readable storage medium provided in this invention achieve the distinction and modeling of moving objects and static backgrounds in dynamic scenes by generating a motion region weight map, a median threshold bitmap, a superpixel region map, and a base weight map. The motion region weight map is corrected using the median threshold bitmap and superpixel region map to improve the spatial consistency and boundary fit of motion detection. Motion interference in the base weight map is removed using the final motion weight map to obtain an accurate fused weight map. Based on this, multi-exposure fusion is completed by combining the original image sequence, which can effectively suppress ghosting artifacts caused by object motion while preserving highlight and shadow details, thus improving the quality and realism of the fused image.

[0046] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart of a multi-exposure image fusion method provided by an embodiment of the present invention is shown.

[0049] Figure 2 This diagram illustrates another flowchart of the multi-exposure image fusion method provided in an embodiment of the present invention.

[0050] Figure 3 A block diagram of a multi-exposure image fusion apparatus provided in an embodiment of the present invention is shown.

[0051] Figure 4 A block diagram of a multi-exposure image fusion apparatus provided in an embodiment of the present invention is shown. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0053] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0054] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0055] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0056] Please refer to Figure 1 , Figure 1 A schematic flowchart of a multi-exposure image fusion method provided by an embodiment of the present invention is shown. The method includes the following steps:

[0057] Step S10: Generate a motion region weight map, a median threshold bitmap, a superpixel region map, and a basic weight map for each original image based on the original image sequence. The motion region weight map represents the location region of dynamic objects and static background in the original image. The median threshold bitmap represents the binarized motion region obtained by median filtering of the original image. The superpixel region map represents the local region division of the original image whose boundary conforms to the object contour. The basic weight map represents the illumination quality and contribution weight of each pixel in the original image.

[0058] In this embodiment of the invention, a motion region weight map, a median threshold bitmap, a superpixel region map, and a basic weight map are generated for each original image in the original image sequence. The motion region weight map is used to initially delineate pixel regions in the original image that may belong to moving objects or static backgrounds; its generation depends on the differences between the current original image and the reference image.

[0059] The principle behind median threshold bitmap generation is to compare the grayscale value of each pixel in each original image in the original image sequence with the corresponding median grayscale value of the original image to generate a binarized image. This processing method is independent of the specific brightness value of the original image, and only related to the median grayscale value, providing a robust motion cue that is unaffected by exposure differences.

[0060] The superpixel region map divides the original image into semantically consistent local regions, enabling subsequent processing to perform consistency judgments on a local region basis. The base weight map is obtained by fusing the original images based on traditional static scene fusion algorithms. The base weight map reflects the comprehensive contribution of each pixel in each original image in terms of lighting rationality, contrast, saturation, etc., and serves as the basic basis for the final fusion.

[0061] Step S20: Correct the corresponding motion region weight map based on all median threshold bitmaps and the superpixel region map of each original image to obtain the final motion weight map corresponding to each original image.

[0062] In this embodiment of the invention, cross-frame consistency information provided by the median threshold bitmap of all original images is used to suppress or enhance false detection regions in the motion region weight map, thereby mitigating motion misjudgment caused by exposure differences. Based on this, and combined with the spatial coherence constraints of the superpixel region map, the correction operation is extended to the superpixel level to generate the final motion weight map, ensuring the continuity and integrity of motion boundaries. In the final motion weight map, an element of 1 indicates that the pixel at that location belongs to the static background, while an element of 0 indicates that the pixel at that location belongs to a dynamic object.

[0063] Step S30: Remove motion interference from the corresponding base weight map based on the final motion weight map corresponding to each original image to obtain the fusion weight map corresponding to each original image.

[0064] In this embodiment of the invention, the basic weight map generated by the traditional static scene fusion algorithm during multi-exposure image fusion may produce incorrect weight allocation in dynamic scenes due to the presence of moving objects, resulting in artifacts such as "ghosting" or abnormal brightness in the fusion result. To solve this problem, motion-aware correction needs to be applied to the basic weight map to eliminate interference from dynamic content.

[0065] By combining the final motion weight map corresponding to each original image with the base weight map, effective masking (i.e., removal of motion regions) is achieved, resulting in a more robust fused weight map. Specifically, the base weight map reflects the ideal contribution of each original image under the assumption of no motion. The expression for the base weight map is:

[0066]

[0067]

[0068]

[0069]

[0070]

[0071] in, It is the base weight map corresponding to the k-th original image in the original image sequence; N is the number of original images in the original image sequence; This is to prevent the denominator from being zero, thus minima. It is the contrast of the k-th original image; It is the saturation of the k-th original image; It is the exposure of the k-th original image; It is the red channel value of the pixel in the kth original image; It is the green channel value of the pixel in the k-th original image; It is the blue channel value of the pixel in the k-th original image; It is the mean of the red, green and blue channels of the pixels in the k-th original image; It is the Laplace operation; It is the grayscale image of the k-th original image; It is a preset constant, such as 0.2.

[0072] Based on this, the base weight map is weighted and suppressed using the final motion weight map to generate a fused weight map, the expression of which is:

[0073]

[0074] in, It is the fusion weight corresponding to the k-th original image; It is the final motion weight map corresponding to the k-th original image.

[0075] It's important to note that after calculating the fusion weight map, an integrity check is necessary to prevent information loss due to excessive culling. Specifically, this checks if a pixel location has a weight of 0 in all original images. This could be caused by misjudging moving regions or excessive dilation, and if left unaddressed, it will create holes or structural breaks in the fused image. Therefore, once a zero-weight case is detected, the weight of that pixel location in the reference image is automatically set to 1 to ensure the spatial continuity and visual integrity of the fusion result.

[0076] Step S40: Fuse the entire fusion weight map and the original image sequence to obtain the fused image.

[0077] In this embodiment of the invention, the fusion weight map guides the degree of participation of each exposure image at different spatial locations, with high-weight regions dominating the construction of the output result, while low-weight regions are weakened or excluded.

[0078] In summary, the multi-exposure image fusion method provided in this invention generates a motion region weight map, a median threshold bitmap, a superpixel region map, and a base weight map to distinguish and model moving objects from static backgrounds in dynamic scenes. The motion region weight map is corrected using the median threshold bitmap and the superpixel region map, improving the spatial consistency and boundary fit of motion detection. Furthermore, motion interference in the base weight map is removed using the final motion weight map, resulting in an accurate fused weight map. Based on this, multi-exposure fusion is performed using the original image sequence, effectively suppressing ghosting artifacts caused by object motion while preserving highlight and shadow details, thus improving the quality and realism of the fused image.

[0079] Optionally, the following is a possible implementation for generating the motion region weight map, median threshold bitmap, superpixel region map, and base weight map for each original image. Figure 1 The sub-steps of step S10 may include:

[0080] Step S100: Select the best-exposed original image from the original image sequence as the reference image.

[0081] In this embodiment of the invention, a reliable benchmark is established for cross-frame comparison by selecting the best-exposed original image from the original image sequence as a reference image. The selection of the reference image is based on the reasonableness of its overall exposure level, and typically an image with a grayscale mean close to the median brightness (e.g., 0.5) is selected to ensure that there is neither significant overexposure nor severe underexposure, and that scene details are well preserved.

[0082] Based on this, the reference image is used for structural and brightness comparisons with other exposed images, providing a stable reference benchmark for motion detection. In other words, by introducing an optimal exposure reference source, the stability and accuracy of subsequent motion estimation processes are improved.

[0083] Step S110: Generate a motion region weight map corresponding to each original image based on the grayscale difference, Euclidean distance, and structural similarity between each original image and the reference image.

[0084] In this embodiment of the invention, by comparing the reference image with each original image in the original image sequence within a local window (i.e., a preset block size), the structural similarity, grayscale difference, and Euclidean distance between the two images are calculated to check and mark the regions that are initially judged to have motion, thus obtaining a motion region weight map corresponding to each original image.

[0085] Step S120: Calculate the grayscale median of each original image, and generate a median threshold bitmap corresponding to each original image based on the grayscale median and the grayscale values ​​of each pixel in the original image.

[0086] After initial detection of motion regions, the preliminary judgment results (i.e., the motion region weight map) are often biased due to differences in exposure, thus affecting the reliability and accuracy of the detection. To solve this problem, a Median Threshold Bitmap (MTB) can be used to correct errors in the motion region weight map.

[0087] It should be understood that the median grayscale value of the original image refers to the average grayscale value of all pixels in the original image. By comparison, pixels with grayscale values ​​higher than the median grayscale value are assigned a weight of 1 in the median threshold bitmap, while pixels with grayscale values ​​no higher than the median grayscale value are assigned a weight of 0. Therefore, the generated median threshold bitmap is not affected by exposure time, but rather depends on the pixel's position relative to the overall image's brightness median (i.e., the grayscale median), exhibiting good cross-image consistency. By introducing a median invariance mechanism, effective support is provided for subsequent correction of motion false detections caused by exposure differences.

[0088] The expression for the median threshold bitmap is:

[0089]

[0090] in, It is the weight of the k-th original image in the pixel (x,y) in the median threshold bitmap; It is the grayscale image of the k-th original image. It is the median gray value of the k-th original image.

[0091] Step S130: Cluster each pixel in each original image to generate a superpixel region map corresponding to each original image.

[0092] In this embodiment of the invention, due to factors such as noise, changes in illumination, or motion blur, the boundaries of moving regions may still have dangerous missed detections or incomplete overlap with the real moving regions. To address this issue, superpixel segmentation technology can be combined to further expand and optimize the detection results. By introducing a segmentation strategy based on color-space joint metric, high-level semantic organization of image content is achieved, laying a structural foundation for fine-grained correction of moving regions, thereby ensuring the integrity and accuracy of moving regions.

[0093] Simple Linear Iterative Clustering (SLIC) is an efficient and widely used superpixel segmentation algorithm. Its core idea is to cluster pixels in the original image using a joint distance metric of color and spatial proximity, thereby generating compact and uniform superpixel regions, resulting in a superpixel region map for each original image. The goal of SLIC is to minimize the distance (i.e., the combined distance) between each pixel and the center of its superpixel, where the distance metric combines color similarity and spatial proximity.

[0094] It should be noted that initially, the original image is divided into multiple superpixel regions, and the superpixel center of each superpixel region is the coordinate of the center position of the superpixel region. The formula for calculating the overall distance D is:

[0095]

[0096]

[0097]

[0098]

[0099] in, It is the distance in the color space, usually using the Euclidean distance in the CIELAB color space; is the Euclidean distance between the center of the pixel and the superpixel on the image plane; m is a preset weight parameter used to balance the relative importance of color distance and spatial distance; S is the initial grid spacing of the superpixel region used to control the size of the superpixel region; It is the Lab value of the pixel; is the Lab value of the superpixel center; N is the total number of pixels in the original image; a is the number of preset superpixel regions.

[0100] Step S140: Perform static scene fusion on each original image to obtain the basic weight map corresponding to each original image.

[0101] In this embodiment of the invention, the basic weight graph refers to the quality evaluation graph generated based on the traditional static fusion criteria, ignoring dynamic content.

[0102] As can be seen, the embodiments of the present invention achieve accurate differentiation between moving objects and static backgrounds in dynamic scenes by generating motion region weight maps, median threshold bitmaps, superpixel region maps, and basic weight maps. Motion regions are detected by combining grayscale differences, Euclidean distance, and structural similarity of reference images. Median threshold bitmaps are used to enhance cross-exposure consistency, and superpixel clustering ensures that region boundaries conform to object contours. Simultaneously, quality assessments of each image are obtained based on static scene fusion, thus providing accurate weight guidance for subsequent fusion and effectively suppressing ghosting artifacts while preserving rich details.

[0103] Optionally, regarding how to generate the motion region weight map, the following is a possible implementation. The sub-steps of step S110 may include:

[0104] Step S111: For each original image, determine the corresponding reference image block in the reference image based on the coordinates of each pixel in the original image and the preset block size.

[0105] In this embodiment of the invention, it is assumed that each pixel in the original image may be part of a dynamic object. For the coordinates (x, y) of each pixel in each original image, a local image block centered on that pixel is defined according to a preset block size (2r+1)×(2r+1), and the corresponding reference image block located at the same coordinate position is extracted in the reference image.

[0106] Step S112: The pixel in the reference image block with the smallest difference in grayscale value from the corresponding pixel in the original image is determined as the target pixel.

[0107] In this embodiment of the invention, the pixel located at (x, y) in the current original image is considered as part of a potential dynamic object. The motion region is detected by finding the pixel with the most similar grayscale value within its corresponding reference image block. The difference between the grayscale value of the pixel (x, y) in the current original image and the grayscale value of each pixel in the corresponding reference image block is calculated, and the pixel with the smallest difference in the reference image block is determined as the target pixel.

[0108] Step S113: Determine the grayscale difference of each pixel in the original image based on the grayscale value of the target pixel and the grayscale value of the corresponding pixel in the original image.

[0109] In this embodiment of the invention, for each pixel (x, y) in the original image, the grayscale difference (i.e., the smallest grayscale difference in the reference image block) indicates that the pixel (x, y) is most similar to the target pixel in the reference image block and is very likely to belong to the same moving object. Grayscale difference The calculation formula is:

[0110]

[0111] in, It is the gray value of the pixel at coordinates (x, y) in the original image; It is the grayscale value of the pixel at coordinates (x+i, y+j) in the reference image patch.

[0112] Step S114: Determine the Euclidean distance for each pixel in the original image based on the pixel coordinates of the target pixel and the corresponding pixel coordinates in the original image.

[0113] In this embodiment of the invention, the spatial displacement relationship between the target pixel and the corresponding pixel in the original image also needs to be considered. The Euclidean distance is calculated based on the actual coordinates (x+i, y+j) of the target pixel in the reference image block and the coordinates (x, y) of the pixel in the original image. Euclidean distance is used to measure the relative offset between two pixels in space. A larger Euclidean distance often indicates significant motion behavior.

[0114] Step S115: Determine the structural similarity of each pixel in the original image based on the image block corresponding to each pixel in the original image and the reference image block.

[0115] In this embodiment of the invention, a structural similarity measurement mechanism is introduced to further improve the robustness of motion detection. The structural similarity between the reference image and each original image is evaluated based on the similarity of image patches. Assuming the reference image... The reference image patch located at (x, y) is The k-th original image The image patch located at (x, y) is Structural similarity The calculation formula is:

[0116]

[0117]

[0118]

[0119]

[0120] in, It is a preset error; It is the gray value of the j-th pixel in the reference image block; It is the gray value of the j-th pixel in the image block corresponding to (x, y) in the k-th original image; It is the average gray value of the reference image patch; It is the average gray value of the image patch corresponding to (x, y) in the k-th original image; , and These are all temporary variables.

[0121] It should be understood that a higher structural similarity indicates smaller differences between structural image patches, while a lower structural similarity indicates larger differences between structural image patches, which may be due to motion, occlusion, or abrupt changes in illumination.

[0122] Step S116: Determine the motion weight corresponding to each pixel in the original image based on the gray-level difference, Euclidean distance, and structural similarity of each pixel in the original image; construct the motion region weight map corresponding to the original image from the motion weights corresponding to all pixels in the original image.

[0123] In this embodiment of the invention, motion weights for each pixel are generated by fusing grayscale differences, Euclidean distance, and structural similarity. The motion weight calculation formula for a pixel located at (x, y) is as follows:

[0124]

[0125]

[0126] in, It is the motion weight of the pixel located at coordinates (x, y) in the k-th original image. A value of 0 indicates that the pixel located at (x, y) belongs to the moving region, and a value of 1 indicates that the pixel located at (x, y) belongs to the static region. It is the difference in the mean gray level between the reference image patch corresponding to coordinates (x, y) and the image patch corresponding to coordinates (x, y) in the k-th original image; It is a temporary variable.

[0127] Optionally, regarding how to generate the final motion weight map corresponding to each original image, the following is a possible implementation. Please refer to... Figure 2 , Figure 1 The sub-steps of step S20 may include:

[0128] Step S200: Generate a global median bitmap based on all median threshold bitmaps.

[0129] In this embodiment of the invention, when the value of the same coordinate is the same in all median threshold bitmaps, the weight of the same coordinate in the global median bitmap is set to 1; otherwise, it is set to 0. The weight of the coordinate located at (x, y) in the global median bitmap... The calculation formula is:

[0130]

[0131] Step S210: Generate an intermediate motion weight map corresponding to each original image based on the global median bitmap and the motion region weight map corresponding to each original image.

[0132] In this embodiment of the invention, the global median bitmap is used as a mask to correct the motion region weight map corresponding to each original image. This ensures that the motion weight of a pixel is retained or enhanced only when it shows signs of motion in the global view and is also identified as motion in the current frame. Meanwhile, regions that are not identified as motion in the global scope are suppressed, effectively reducing false positives caused by changes in lighting or mismatched textures and improving the accuracy of motion detection.

[0133] Step S220: Generate the final motion weight map corresponding to each original image based on the superpixel region map and the corresponding intermediate motion weight map corresponding to each original image.

[0134] In this embodiment of the invention, to further optimize the accuracy and robustness of motion detection, a spatial homogeneity constraint for superpixel regions is introduced, transforming pixel-level motion judgment into region-level consistency analysis to address local texture interference and detection jitter issues. The weights located at (x, y) in the final motion weight map corresponding to the k-th original image are... The formula is determined as follows:

[0135]

[0136] in, Superpixel region The number of regions with a weight of 1 in the intermediate motion weight map; Superpixel region The number of pixels it contains; The meaning of "and" is that (x, y) belongs to the superpixel region. At the same time, it is also necessary to satisfy the requirement that the number of regions with a weight of 1 in the corresponding region of the intermediate motion weight map is equal to the number of superpixel regions. If the proportion of the number of pixels contained is less than the preset static threshold, the superpixel region is considered to be in a static state, and the final motion weight of the corresponding pixel in the superpixel region is set to 1; otherwise, the superpixel region is considered to contain dynamic objects and is set to 0; b is the preset static threshold, for example, 0.1.

[0137] As can be seen, the embodiments of the present invention effectively enhance the spatial consistency and noise resistance of motion region judgment by quantitatively evaluating the motion response within each superpixel region and making binarization decisions based on a preset static threshold. This region-based statistical strategy not only preserves the integrity of the object contour but also reduces the impact of isolated noise points or falsely detected pixels on the fusion process, thereby improving the adaptability and stability of the multi-exposure image fusion method in complex dynamic scenes.

[0138] Alternatively, one possible implementation for generating the intermediate motion weight map corresponding to each original image is provided below. Figure 2 The sub-steps of step S210 may include:

[0139] Step S211: The global median bitmap is processed using opening and closing operations to obtain the processed global median bitmap.

[0140] In this embodiment of the invention, the global median bitmap is used to characterize the bright and dark structure pattern that is consistent throughout the entire original image sequence. However, due to sudden changes in illumination, noise interference, or ghosting and edge blurring caused by fast-moving objects, isolated noise points, small misjudged areas, or broken segments may exist in the global median bitmap, causing the outline of the moving object to be segmented into multiple discontinuous parts, thereby reducing the ghosting removal capability in the subsequent fusion process.

[0141] To address the aforementioned issues, morphological opening and closing operations are used to process the global median bitmap to remove noise, fill holes, and connect broken regions, resulting in a processed global median bitmap that improves the integrity and robustness of the moving region. The opening operation first erodes and then dilates the global median bitmap to remove small objects and noise while preserving larger structures. The expression for the opening operation is as follows:

[0142]

[0143] in, It is a pre-defined structural element; The erosion operation represents the shrinking of the target region in the global median bitmap using a structuring element. The expansion operation uses a structuring element to expand the target region in the global median bitmap.

[0144] Building upon this, the closing operation is further used to fill small voids or cracks that still exist after the opening operation. The expression for the closing operation is as follows:

[0145]

[0146] in, It is the global median bitmap after the opening operation.

[0147] It should be noted that the definitions of expansion and corrosion operations are as follows:

[0148]

[0149]

[0150]

[0151] Here, A represents the image that has been subjected to dilation or erosion operations.

[0152] Step S212: Perform pixel-by-pixel multiplication based on the motion region weight map corresponding to each original image and the processed global median bitmap to obtain a temporary motion weight map corresponding to each original image.

[0153] In this embodiment of the invention, the global median bitmap after the closing operation is the processed global median bitmap. The processed global median bitmap is used to correct the motion region in the motion region weight map corresponding to each original image, resulting in a temporary motion weight map for each original image. The calculation formula for the temporary motion weight map is:

[0154]

[0155] in, These are the temporary motion weights corresponding to the k-th original image; It is the processed global median bitmap.

[0156] Step S213: Dilate the temporary motion weight map corresponding to each original image to obtain the intermediate motion weight map corresponding to each original image.

[0157] In this embodiment of the invention, due to the influence of factors such as changes in lighting, the complexity of object textures, and the speed of motion during motion detection, the detected motion area may not completely cover the actual range of the moving object, resulting in partial omissions or incomplete boundaries in the edge areas of the moving object.

[0158] To address this issue, a morphological dilation operation is performed on the temporary motion weight map corresponding to each original image. The dilation operation expands the motion region using structuring elements, effectively filling in missing parts of the moving object's edges and smoothing the boundaries of the motion region to better fit the object's true contour, thus obtaining an intermediate motion weight map corresponding to each original image.

[0159] Optionally, regarding how to perform the expansion process, the following is a possible implementation. The sub-steps of step S213 may include:

[0160] Step S213-1: Determine the total number of pixels and the number of moving pixels for each original image based on the temporary motion weight map corresponding to each original image.

[0161] In this embodiment of the invention, the total number of pixels is the total number of pixels in the original image. The number of moving pixels is obtained by counting the number of pixels marked as moving regions in the motion region weight map, typically represented by pixels with a weight value of 0.

[0162] Step S213-2: Determine the number of dilation steps for each original image based on the total number of pixels and the number of moving pixels corresponding to each original image.

[0163] In this embodiment of the invention, the formula for calculating the number of expansions is:

[0164]

[0165] Where H is the height of the original image; W is the width of the original image; S is a preset scaling factor used to control the sensitivity of the dilation count; and r is the proportion of moving pixels to the total number of pixels. This represents rounding down to the nearest integer.

[0166] Step S213-3: Dilate the temporary motion weight map corresponding to each original image according to the number of dilation steps, and obtain the intermediate motion weight map corresponding to each original image.

[0167] In this embodiment of the invention, the number of dilation steps is dynamically adjusted according to the image content, reflecting the ability to respond differently to motion features in different scenes. Based on this, the corresponding temporary motion weight map is subjected to successive morphological dilation processing according to the number of dilation steps corresponding to each original image, and finally an intermediate motion weight map that can effectively cover the true range of the motion region is obtained.

[0168] Alternatively, one possible implementation for generating the fused image is provided below. Figure 1 The sub-steps of step S40 may include:

[0169] Step S400: Construct a Gaussian pyramid based on each fusion weight graph.

[0170] Step S410: Construct the Laplacian pyramid based on the original image sequence.

[0171] In this embodiment of the invention, a set of layered representations from coarse to fine is generated by performing multiple downsampling operations on each fused weight map, with each layer corresponding to the weight distribution at different spatial resolutions. Wherein, the first... The fusion weight map of the layers is obtained after Gaussian filtering and downsampling. This serves as a guide for information fusion strategies at this scale. Similarly, a Laplacian pyramid is constructed based on the original image sequence.

[0172] Step S420: Perform layer-by-layer fusion based on the Gaussian pyramid and the Laplace pyramid, and determine the bottom layer image as the fused image.

[0173] In this embodiment of the invention, the Gaussian pyramid and the Laplace pyramid are fused layer by layer. The core of the fusion lies in the treatment of each scale. The mathematical expression for performing a weighted summation operation is as follows:

[0174]

[0175] in, It is the first Laplacian pyramid decomposition of the fused image. Layer representation; It is the first of the Laplacian pyramid decompositions of the original image. Layer representation; It is the first Gaussian pyramid decomposition of the fused weighted graph. Layer representation.

[0176] After completing the weighted calculations at all levels, the image reconstruction stage begins. Starting from the highest layer (the coarsest layer), the image resolution is gradually restored upwards. This is done recursively, meaning that the image after fusion at each layer... It is obtained by adding the Laplacian representation of the current scale to the upsampled version of the reconstruction result from the previous layer, as shown in the following formula:

[0177]

[0178] in, Indicates the first pyramid The image after layer reconstruction; This is an upsampling operation. The layer 0 (i.e., the bottom layer) corresponds to... This is a fused image.

[0179] Based on the same inventive concept, the basic principle and technical effects of the multi-exposure image fusion device provided in this embodiment are the same as those in the above embodiments. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments.

[0180] Please refer to Figure 3 , Figure 3 This is a block diagram of a multi-exposure image fusion apparatus 400 provided in an embodiment of the present invention. The multi-exposure image fusion apparatus 400 includes a processing module 410, a correction module 420, and a fusion module 430.

[0181] The processing module 410 is used to generate a motion region weight map, a median threshold bitmap, a superpixel region map, and a basic weight map for each original image based on the original image sequence. The motion region weight map represents the location region where the dynamic object and static background are located in the original image. The median threshold bitmap represents the binarized motion region obtained by median filtering of the original image. The superpixel region map represents the local region division in the original image whose boundary conforms to the object contour. The basic weight map represents the illumination quality and contribution weight of each pixel in the original image.

[0182] The correction module 420 is used to correct the corresponding motion region weight map based on the total median threshold bitmap and the superpixel region map of each original image, so as to obtain the final motion weight map corresponding to each original image.

[0183] The fusion module 430 is used to remove motion interference from the corresponding basic weight map based on the final motion weight map corresponding to each original image to obtain the fusion weight map corresponding to each original image; and to fuse all the fusion weight maps and the original image sequence to obtain the fused image.

[0184] In summary, the multi-exposure image fusion apparatus provided in this embodiment of the invention achieves the distinction and modeling of moving objects and static backgrounds in dynamic scenes by generating a motion region weight map, a median threshold bitmap, a superpixel region map, and a base weight map. The motion region weight map is corrected using the median threshold bitmap and the superpixel region map to improve the spatial consistency and boundary fit of motion detection. Furthermore, motion interference in the base weight map is removed using the final motion weight map, resulting in an accurate fused weight map. Based on this, multi-exposure fusion is completed by combining the original image sequence, effectively suppressing ghosting artifacts caused by object motion while preserving highlight and shadow details, thus improving the quality and realism of the fused image.

[0185] Optionally, the processing module 410 is specifically used to select the best-exposed original image from the original image sequence as a reference image; generate a motion region weight map corresponding to each original image based on the gray-level difference, Euclidean distance, and structural similarity between each original image and the reference image; calculate the gray-level median of each original image, and generate a median threshold bitmap corresponding to each original image based on the gray-level median and the gray-level values ​​of each pixel in the original image; cluster each pixel in each original image to generate a superpixel region map corresponding to each original image; and perform static scene fusion on each original image to obtain a basic weight map corresponding to each original image.

[0186] Optionally, the processing module 410 is specifically configured to, for each original image, determine a corresponding reference image block in the reference image based on the coordinates of each pixel in the original image and a preset block size; determine the pixel in the reference image block with the smallest grayscale value difference from the corresponding pixel in the original image as the target pixel; determine the grayscale difference corresponding to each pixel in the original image based on the grayscale value of the target pixel and the grayscale value of the corresponding pixel in the original image; determine the Euclidean distance corresponding to each pixel in the original image based on the pixel coordinates of the target pixel and the corresponding pixel coordinates in the original image; determine the structural similarity corresponding to each pixel in the original image based on the image block corresponding to each pixel in the original image and the reference image block; determine the motion weight corresponding to each pixel in the original image based on the grayscale difference, Euclidean distance, and structural similarity; and construct a motion region weight map corresponding to the original image from the motion weights corresponding to all pixels in the original image.

[0187] Optionally, the correction module 420 is specifically used to generate a global median bitmap based on all median threshold bitmaps; generate an intermediate motion weight map corresponding to each original image based on the global median bitmap and the motion region weight map corresponding to each original image; and generate a final motion weight map corresponding to each original image based on the superpixel region map corresponding to each original image and the corresponding intermediate motion weight map.

[0188] Optionally, the correction module 420 is specifically used to process the global median bitmap using opening and closing operations to obtain a processed global median bitmap; to perform pixel-by-pixel multiplication based on the motion region weight map corresponding to each original image and the processed global median bitmap to obtain a temporary motion weight map corresponding to each original image; and to dilate the temporary motion weight map corresponding to each original image to obtain an intermediate motion weight map corresponding to each original image.

[0189] Optionally, the correction module 420 is specifically used to determine the total number of pixels and the number of moving pixels corresponding to each original image based on the temporary motion weight map corresponding to each original image; determine the dilation number corresponding to each original image based on the total number of pixels and the number of moving pixels corresponding to each original image; and dilate the corresponding temporary motion weight map according to the dilation number corresponding to each original image to obtain the intermediate motion weight map corresponding to each original image.

[0190] Optionally, the fusion module 430 is used to construct a Gaussian pyramid based on each fusion weight map; construct a Laplacian pyramid based on the original image sequence; perform layer-by-layer fusion based on the Gaussian pyramid and the Laplacian pyramid, and determine the bottom layer image as the fused image.

[0191] Please refer to Figure 4This is a block diagram illustrating an electronic device 500 provided in an embodiment of the present invention. The electronic device 500 includes, but is not limited to, a personal computer (PC), a personal digital assistant (PDA), a laptop computer, a tablet computer, and a server. The electronic device 500 includes a memory 510, a processor 520, and a communication module 530. The memory 510, processor 520, and communication module 530 are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0192] The memory 510 is used to store programs or data. The memory 510 may be, but is not limited to, random access memory, read-only memory, programmable read-only memory, erasable read-only memory, electrically erasable read-only memory, etc.

[0193] The processor 520 is used to read / write data or programs stored in the memory 510 and perform corresponding functions. For example, when a computer program stored in the memory 510 is executed by the processor 520, the multi-exposure image fusion method disclosed in the above embodiments can be implemented.

[0194] The communication module 530 is used to establish a communication connection between the electronic device 500 and other communication terminals via a network, and to send and receive data via the network.

[0195] It should be understood that, Figure 4 The structure shown is only a schematic diagram of the electronic device 500. The electronic device 500 may also include components that are larger than those shown. Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown. Figure 4 The components shown can be implemented using hardware, software, or a combination thereof.

[0196] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor 520, implements the multi-exposure image fusion method disclosed in the above embodiments.

[0197] This invention also provides a program product that, when executed by processor 520, implements the multi-exposure image fusion method disclosed in the above embodiments.

[0198] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0199] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0200] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0201] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-exposure image fusion method, characterized in that, The method includes: Based on the original image sequence, a motion region weight map, a median threshold bitmap, a superpixel region map, and a basic weight map are generated for each original image. The motion region weight map represents the location region where dynamic objects and static backgrounds are located in the original image. The median threshold bitmap represents the binarized motion region obtained by median filtering of the original image. The superpixel region map represents the local region division in the original image whose boundary conforms to the object contour. The basic weight map represents the illumination quality and contribution weight of each pixel in the original image. The motion region weight map is corrected based on all the median threshold bitmaps and the superpixel region map of each original image to obtain the final motion weight map corresponding to each original image. Based on the final motion weight map corresponding to each original image, the motion interference of the corresponding basic weight map is removed to obtain the fusion weight map corresponding to each original image; The fused image is obtained by fusing all the fusion weight maps and the original image sequence.

2. The multi-exposure image fusion method according to claim 1, characterized in that, The step of generating a motion region weight map, a median threshold bitmap, a superpixel region map, and a basic weight map for each original image based on the original image sequence includes: Select the best-exposed original image from the original image sequence as the reference image; A motion region weight map corresponding to each original image is generated based on the grayscale difference, Euclidean distance, and structural similarity between each original image and the reference image. Calculate the median grayscale value of each original image, and generate a median threshold bitmap corresponding to each original image based on the median grayscale value and the grayscale value of each pixel in the original image; Cluster each pixel in each of the original images to generate a superpixel region map corresponding to each of the original images; Static scene fusion is performed on each original image to obtain the basic weight map corresponding to each original image.

3. The multi-exposure image fusion method according to claim 2, characterized in that, The step of generating a motion region weight map corresponding to each original image based on the grayscale difference, Euclidean distance, and structural similarity between each original image and a reference image includes: For each of the original images, a corresponding reference image block is determined in the reference image based on the coordinates of each pixel in the original image and a preset block size; The pixel with the smallest grayscale value difference between the reference image block and the corresponding pixel in the original image is determined as the target pixel; The grayscale difference of each pixel in the original image is determined based on the grayscale value of the target pixel and the grayscale value of the corresponding pixel in the original image; The Euclidean distance for each pixel in the original image is determined based on the pixel coordinates of the target pixel and the corresponding pixel coordinates in the original image. The structural similarity of each pixel in the original image is determined based on the image patch corresponding to each pixel in the original image and the reference image patch; The motion weights corresponding to each pixel in the original image are determined based on the grayscale difference, Euclidean distance, and structural similarity of each pixel in the original image; the motion weights corresponding to all pixels in the original image constitute the motion region weight map corresponding to the original image.

4. The multi-exposure image fusion method according to claim 1, characterized in that, The step of correcting the corresponding motion region weight map based on all the median threshold bitmaps and the superpixel region map of each original image to obtain the final motion weight map corresponding to each original image includes: Generate a global median bitmap based on all the aforementioned median threshold bitmaps; Based on the global median bitmap and the motion region weight map corresponding to each original image, an intermediate motion weight map corresponding to each original image is generated; Based on the superpixel region map and the corresponding intermediate motion weight map of each original image, a final motion weight map is generated for each original image.

5. The multi-exposure image fusion method according to claim 4, characterized in that, The step of generating an intermediate motion weight map corresponding to each original image based on the global median bitmap and the motion region weight map corresponding to each original image includes: The global median bitmap is processed using opening and closing operations to obtain the processed global median bitmap; A temporary motion weight map corresponding to each original image is obtained by performing a pixel-by-pixel multiplication based on the motion region weight map corresponding to each original image and the processed global median bitmap. Dilatation is performed on the temporary motion weight map corresponding to each original image to obtain the intermediate motion weight map corresponding to each original image.

6. The multi-exposure image fusion method according to claim 5, characterized in that, The step of dilating the temporary motion weight map corresponding to each of the original images to obtain the intermediate motion weight map corresponding to each of the original images includes: The total number of pixels and the number of moving pixels corresponding to each original image are determined based on the temporary motion weight map corresponding to each original image; The number of dilation steps for each original image is determined based on the total number of pixels and the number of moving pixels corresponding to each original image; By performing dilation processing on the temporary motion weight map corresponding to the dilation number of each original image, an intermediate motion weight map corresponding to each original image is obtained.

7. The multi-exposure image fusion method according to claim 1, characterized in that, The step of fusing the entire fusion weight map and the original image sequence to obtain a fused image includes: Construct a Gaussian pyramid based on the fusion weight graphs described above; Construct the Laplacian pyramid based on the original image sequence; The Gaussian pyramid and the Laplace pyramid are merged layer by layer, and the bottom layer image is determined as the merged image.

8. A multi-exposure image fusion device, characterized in that, The device includes: The processing module is used to generate a motion region weight map, a median threshold bitmap, a superpixel region map, and a basic weight map for each original image based on the original image sequence. The motion region weight map represents the location regions of dynamic objects and static backgrounds in the original image. The median threshold bitmap represents the binarized motion region obtained by median filtering of the original image. The superpixel region map represents the local region division in the original image whose boundaries conform to the object contours. The basic weight map represents the illumination quality and contribution weight of each pixel in the original image. The correction module is used to correct the corresponding motion region weight map based on all the median threshold bitmaps and the superpixel region map of each original image, so as to obtain the final motion weight map corresponding to each original image. The fusion module is used to remove motion interference from the corresponding base weight map based on the final motion weight map corresponding to each original image to obtain a fused weight map corresponding to each original image; and to fuse all the fused weight maps and the original image sequence to obtain a fused image.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program that can be executed by the processor to implement the multi-exposure image fusion method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the multi-exposure image fusion method as described in any one of claims 1-7.

Images