A High Dynamic Range Imaging Method Based on Multi-Scale Progressive Reconstruction Networks

By using the multi-scale progressive reconstruction network MPRNet, combined with multi-scale progressive reconstruction and dual-stream exposure recovery modules, the problems of ghosting, blurring, and distortion under large-scale motion and extreme exposure are solved, achieving high-quality HDR imaging.

CN118799230BActive Publication Date: 2026-11-17NORTHWEST NORMAL UNIVERSITY
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410639970.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-22
Publication Date
2026-11-17
Estimated Expiration
2044-05-22

AI Technical Summary

Technical Problem

Existing high dynamic range imaging technologies are prone to ghosting, blurring, and distortion in large-scale motion and extreme exposure scenarios, making it difficult to achieve high-quality HDR imaging.

Method used

The multi-scale progressive reconstruction network MPRNet is adopted, which includes an encoder module, a multi-scale progressive reconstruction module MSPRM, and a dual-stream exposure restoration module DERM. By combining multi-scale progressive reconstruction and global context modeling with dual-stream exposure restoration and content reconstruction, the imaging quality is optimized using a joint supervised loss function.

Benefits of technology

It effectively suppresses motion ghosting, restores overexposed areas, reconstructs rich texture details, and improves HDR imaging quality in dynamic scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118799230B_ABST
    Figure CN118799230B_ABST
Patent Text Reader

Abstract

This invention belongs to the field of high dynamic range imaging technology, and particularly relates to a high dynamic range imaging method based on a multi-scale progressive reconstruction network (MPRNet). The method includes: constructing a multi-scale progressive reconstruction network MPRNet, comprising an encoder module, a multi-scale progressive reconstruction module (MSPRM), a dual-stream exposure recovery module (DERM), and a decoder module; the encoder module is used for feature extraction to obtain corresponding feature maps E1, E2, and E3; the MSPRM is used for multi-scale progressive reconstruction and global context modeling of feature maps E1, E2, and E3 to obtain fused features; the DERM is used for exposure recovery of overexposed areas on the fused features to obtain exposure recovery features F. expo It is also used to complete and repair the content details of the fused features, resulting in the content reconstruction feature F. rec Then, adaptive weighting is performed to obtain the optimized recovery feature F. fused The decoder is used to optimize the recovery of feature F. fused This invention, which uses decoding to predict HDR images, can solve problems such as ghosting, blurring, and distortion in large-scale motion and extreme exposure scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of high dynamic range imaging technology, and particularly relates to a high dynamic range imaging method based on a multi-scale progressive reconstruction network. Background Technology

[0002] High Dynamic Range (HDR) imaging aims to generate images with a wider dynamic range and more realistic details by fusing multiple low dynamic range (LDR) images with different exposures. HDR imaging technology has wide applications in computational photography, computer vision, and graphics, such as scene understanding, photo rendering, and image and video editing. However, in actual shooting, due to camera shake and the presence of moving objects, LDR images are often not perfectly aligned, resulting in ghosting artifacts in the reconstructed HDR results, which seriously affects the visual presentation. Therefore, how to achieve high-quality HDR imaging without ghosting in dynamic scenes has become an urgent problem to be solved.

[0003] Existing mainstream HDR ghosting removal methods are mainly divided into two categories: alignment-based methods and learning-based methods.

[0004] Alignment-based methods align LDR images using global transformations (such as homography) or local transformations (such as optical flow) before fusing them into HDR. However, accurate alignment in complex dynamic scenes is inherently challenging, especially in areas where exposure saturation causes information loss. Furthermore, over-alignment in the image spatial domain can also lead to loss of detail and texture.

[0005] Learning-based methods utilize deep neural networks in a data-driven manner to obtain feature representations for ghosting removal and fusion strategies. These methods alleviate alignment problems to some extent and achieve good performance. However, when faced with large-scale motion and overexposure, existing methods still produce HDR results with significant quality defects such as ghosting, blurring, and distortion. The reasons for this are mainly twofold: 1) the receptive field of convolutional operations is limited, making it difficult to model long-distance dependencies to handle large-scale motion; 2) the lack of special processing for extremely exposed areas leads to irrecoverable overexposure.

[0006] Therefore, how to solve the problems of ghosting, blurring, and distortion caused by large-scale motion and extreme exposure scenes, and achieve high-quality HDR imaging in dynamic scenes, has become an urgent problem to be solved. Summary of the Invention

[0007] To address the shortcomings of the existing technologies, this invention provides a high dynamic range imaging method based on a multi-scale progressive reconstruction network, which can solve problems such as ghosting, blurring, and distortion caused by large-scale motion and extreme exposure scenes, and achieve high-quality HDR imaging in dynamic scenes.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] A high dynamic range imaging method based on a multi-scale progressive reconstruction network includes the following steps:

[0010] S1. Construct the multi-scale progressive reconstruction network MPRNet, including an encoder module, a multi-scale progressive reconstruction module MSPRM, a dual-stream exposure recovery module DERM, and a decoder module.

[0011] The encoder module is used to extract features from the three LDR images with different exposure levels, respectively, to obtain the corresponding feature maps E1, E2, and E3.

[0012] The Multi-Scale Progressive Reconstruction Module (MSPRM) is used to perform multi-scale progressive reconstruction and global context modeling on feature maps E1, E2, and E3 to obtain fused features.

[0013] The dual-stream exposure recovery module (DERM) is used for fusion features. Exposure recovery of the overexposed area was performed, and the exposure recovery feature F was obtained. expo It is also used for fusion features. The content details are supplemented and repaired to obtain the content reconstruction feature F. rec It is also used for the exposure recovery feature F. expo and content reconstruction features F rec Adaptive weighting is performed to obtain the optimized recovery feature F. fused ;

[0014] The decoder is used to optimize the recovery feature F fused Decode the image to obtain the HDR image prediction.

[0015] S2. Train the constructed multi-scale progressive reconstruction network MPRNet;

[0016] S3. Use the trained multi-scale progressive reconstruction network MPRNet for actual high dynamic range imaging processing.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] This method uses a constructed multi-scale progressive reconstruction network (MPRNet) for high dynamic range imaging. During high dynamic range imaging, the MSPRM module performs multi-scale progressive reconstruction and global context modeling on feature maps E1, E2, and E3 to obtain fused features. This progressive fusion approach, spanning scales and exposures, effectively suppresses motion ghosting, providing high-quality feature representations for subsequent refined restoration. The dual-stream exposure recovery module (DERM) performs exposure restoration and content reconstruction, using fused features from both methods. This dual-stream structure and joint supervision better recover overexposed areas while reconstructing rich texture details, improving HDR imaging quality. This setup effectively ensures high-quality HDR imaging in dynamic scenes.

[0019] In summary, this invention can solve problems such as ghosting, blurring, and distortion caused by large-scale motion and extreme exposure scenes, and achieve high-quality HDR imaging in dynamic scenes.

[0020] Preferably, the workflow of the multi-scale progressive refactoring module MSPRM includes:

[0021] First, a feature pyramid with three scales is created through downsampling.

[0022] Next, for the i-th feature at the s-th scale in the t-th stage... CE module embedded in context Perform global context modeling and output enhanced features.

[0023]

[0024] Where NL(·) represents context modeling, which aggregates global context information by calculating the correlation between any two locations;

[0025] Then, the scale selection module (SSM) fuses features from the same and higher levels into the current scale, resulting in cross-scale fused features.

[0026] Then, a separable large kernel attention mechanism is used to fuse features at the same level to obtain the feature representation for the next stage.

[0027] Then, The finely aligned output features are obtained by sequentially passing through a CE module, an SSM module, and a convolutional attention block CA.

[0028]

[0029] Finally, the output features at the original scale are... It is passed to the dual-stream exposure recovery module DERM.

[0030] Thus, the output features Encoding multi-scale motion information from local to global levels provides rich feature representations for high-quality HDR image reconstruction. MSPRM introduces a context embedding module at each scale of the feature pyramid to model global dependencies. Through a scale selection mechanism, it fuses semantic information from the same and higher levels to the current scale. Then, through attention, it adaptively weights different exposure features within the same scale, maximizing the use of complementary information between multi-scale and multi-frame exposure images. This progressive fusion approach across scales and exposures effectively suppresses motion ghosting, providing high-quality feature representations for subsequent fine-grained restoration.

[0031] Preferably, the input of the SSM includes an upsampling branch and a fusion branch; the upsampling branch combines high-level features Mapping to the s-th scale to obtain features Merging branches will combine sibling features The outputs of the two branches are passed to the output; the outputs of the two branches are fused through the SSM module to obtain the cross-scale fused features.

[0032]

[0033]

[0034] Preferably, when the SSM module fuses the outputs of the two branches, it obtains the feature map through global average pooling. The global response is obtained by performing a 1×1 convolution and softmax to obtain nonlinear mapping weight maps at two different scales.

[0035] in, This represents element-wise multiplication.

[0036] In this way, motion priors can be extracted from higher-level semantic features through this vertical propagation, guiding feature selection to adaptively adjust the contribution of features at different scales.

[0037] Preferably, the DERM includes an exposure recovery sub-network for exposure recovery and a content reconstruction sub-network for content reconstruction; the exposure recovery sub-network and the content reconstruction sub-network have the same structure and share the same feature encoder.

[0038] This symmetrical design allows the network to treat exposure recovery and content reconstruction as equally important tasks, and the separate two-stream design helps decouple the complexity of a single learning task.

[0039] Preferably, the working process of the dual-stream exposure recovery module (DERM) includes:

[0040] First, the features are fused using a convolutional block. Compression was performed, reducing the number of channels from 3n feat Reduced to n feat The encoded feature F is obtained. E :

[0041]

[0042] After that, f E Simultaneously, the data is fed into the exposure recovery subnetwork and the content reconstruction subnetwork, respectively, for exposure recovery and content reconstruction, to obtain the exposure recovery feature F. expo and content reconstruction features F rec Among them, both exposure recovery from the network and content reconstruction from the network are RLFN networks;

[0043] F expo =RLFN(F E );

[0044] F rec =RLFN(F E );

[0045] Wherein, RLFN(·) represents the RLFN subnetwork;

[0046] Then, another convolutional block is used to refine the exposure recovery feature F. expo and content reconstruction features F rec Adaptive weighting is performed to obtain the optimized recovery feature F. fused :

[0047] F fused =Conv(F expo ,F rec ).

[0048] In this setup, the dual-stream exposure recovery module (DERM) performs exposure recovery and content reconstruction simultaneously through two symmetrical sub-networks, and uses convolutional blocks to adaptively fuse features from both branches. This dual-stream structure can better recover overexposed areas while reconstructing rich texture details, thus improving HDR image quality.

[0049] Preferably, the decoder module includes decoder 1 and decoder 2;

[0050] Decoder 1 is used to convert F through a 3×3 convolutional layer and a sigmoid function. fused Mapping back to the original RGB space yields the final HDR prediction.

[0051]

[0052] Decoder 2 is used for F expoA 1×1 convolution and sigmoid activation were performed to obtain the exposure recovery prediction.

[0053]

[0054] By setting up an exposure recovery branch as an auxiliary supervisor, the overexposed areas can be better recovered through joint supervision, while reconstructing rich texture details.

[0055] Preferably, the loss function of the multi-scale progressive reconstruction network MPRNet is... for:

[0056]

[0057] Where α and β control the weights of exposure loss and perception loss, respectively; reconstruction loss Exposure loss and perceived loss These correspond to the content reconstruction branches of the multi-scale progressive reconstruction network. Exposure recovery branch And the perceptual quality of the reconstructed image.

[0058] Preferably, reconstruction loss for:

[0059]

[0060]

[0061] Among them, I H For Ground Truth HDR images; μ is a preset fixed value;

[0062] Exposure loss for:

[0063]

[0064]

[0065] Where ε(x) is the exposure feature map calculation function; w y = [0.299, 0.587, 0.114], representing the RGB to luminance (Y) conversion matrix; Let σ be a two-dimensional Gaussian kernel with standard deviation σ, where * denotes the convolution operation, and σ * Controlling the bandwidth of Gaussian weighting, ||A|| F Denotes the Frobenius norm of matrix A;

[0066] Perceived loss The loss function based on VGG16 is as follows:

[0067]

[0068] Where, φ i (·) represents the output of the i-th feature block of VGG16; λ i The weights are the loss values ​​for each feature block.

[0069] In this way, by jointly optimizing reconstruction, exposure, and perceived quality, the loss function can guide the network to generate high-quality HDR images that are free of ghosting, properly exposed, and rich in detail. Attached Figure Description

[0070] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0071] Figure 1 This is a schematic diagram of the structure of the multi-scale progressive reconstruction network MPRNet in this method;

[0072] Figure 2 This is a schematic diagram of the structure of the multi-scale progressive reconfiguration module MSPRM in the embodiment;

[0073] Figure 3 This is a visual comparison chart of the proposed method and mainstream methods in the experimental verification of the embodiments;

[0074] Figure 4 The experimental verification of this invention is shown in the figure below, which compares the results of MPRNet, Kalantari, AHDR, and HDR-GAN on two scenarios of the Senet al. dataset.

[0075] Figure 5 The figure shows the qualitative results of the ablation experiment of MPRNet in the experimental verification of the example. Detailed Implementation

[0076] The following detailed explanation illustrates the specific implementation methods:

[0077] Example:

[0078] This embodiment discloses a high dynamic range imaging method based on a multi-scale progressive reconstruction network, including the following steps:

[0079] S1. Construct the Multi-Scale Progressive Reconstruction Network (MPRNet), including an encoder module, a multi-scale progressive reconstruction module (MSPRM), a dual-stream exposure recovery module (DERM), and a decoder module. In specific implementation, the structure of the MPRNet is as follows: Figure 1 As shown.

[0080] The encoder module is used to extract features from the three LDR images with different exposure levels, respectively, to obtain the corresponding feature maps E1, E2, and E3.

[0081] In practice, the LDR inputs L1, L2, L3 and their corresponding HDR mappings H1, H2, H3 are concatenated by channel to form a 6-channel input, which is then fed into three pseudo-twin encoders. i Extracting initial features, where Encoder i It consists of a 3×3 convolutional layer that maps the input image to a 64-channel feature map.

[0082] E i =Encoder i ([L i H i ]), i = 1, 2, 3.

[0083] The Multi-Scale Progressive Reconstruction Module (MSPRM) is used to perform multi-scale progressive reconstruction and global context modeling on feature maps E1, E2, and E3 to obtain fused features.

[0084] The core idea of ​​the Multi-Scale Progressive Reconstruction Module (MSPRM) is to perform multi-stage adaptive feature filtering and fusion at various scale levels of the feature pyramid to suppress motion ghosting.

[0085] Specifically, for the i-th feature at the s-th scale in the t-th stage:

[0086]

[0087] Specifically, when t = s = 1,

[0088] First, MSPRM creates a feature pyramid with three scales through downsampling. ( Figure 2 (Blue arrow in the middle) uses Context Embedding (CE) to... Perform global context modeling, such as Figure 2 As shown by the yellow arrow, the CE module borrows from the design of GCNet, aggregating information from all locations in the feature map through a self-attention mechanism to capture long-range dependencies. The CE module outputs the enhanced features.

[0089]

[0090] NL(·) represents context modeling, which aggregates global context information by calculating the correlation between any two locations.

[0091] Next, MSPRM builds a scale selection module (SSM) on top of kernel select attention (KSA), such as... Figure 2 As shown, features from the same and higher-level (low-resolution) layers are fused to the current scale. The input of SSM contains an upsampling branch and a fusion branch; the upsampling branch corresponds to the red arrow in the diagram. The upsampling branch fuses features from higher-level layers... Mapping to the s-th scale to obtain features The fusion branch directly combines features of the same level. Passed to the output.

[0092] The outputs of the two branches are fused using the SSM module to obtain cross-scale fused features.

[0093]

[0094]

[0095] in This represents element-wise multiplication. Specifically, feature maps are obtained through global average pooling. The global response is obtained by performing a 1×1 convolution and softmax to obtain nonlinear mapping weight maps at two different scales. This vertical propagation allows for the extraction of motion priors from higher-level semantic features, guiding feature selection to adaptively adjust the contribution of features at different scales. After completing cross-scale fusion, MSPRM uses a large separable kernel attention mechanism to fuse features at the same level. This deep separable kernel attention mechanism expands the receptive field of attention while minimizing computation, helping different features at the same scale to adaptively fuse and obtain the feature representation for the next stage.

[0096] After a fusion Then, by sequentially passing through the CE module, SSM module, and convolutional attention block CA, finely aligned output features are obtained.

[0097]

[0098] Finally, MSPRM will output features at the original scale. These features are passed to the subsequent dual-stream exposure recovery module for exposure restoration and detail repair. They encode multi-scale motion information from local to global perspectives, providing rich feature representations for high-quality HDR image reconstruction.

[0099] The dual-stream exposure recovery module (DERM) is used for fusion features. Exposure recovery of the overexposed area was performed, and the exposure recovery feature F was obtained. expo It is also used for fusion features. The content details are supplemented and repaired to obtain the content reconstruction feature F. rec It is also used for the exposure recovery feature F. expo and content reconstruction features F rec Adaptive weighting is performed to obtain the optimized recovery feature F. fused ;

[0100] Fusion features After multi-scale progressive reconstruction, it already contains rich multi-scale motion and exposure information from local to global perspectives. The next goal is to repair overexposed areas and reconstruct detailed textures, such as... Figure 1 As shown, this invention designs a dual-stream exposure restoration module (DERM).

[0101] DERM comprises two structurally identical subnetworks for Exposure Restoration and Content Reconstruction, respectively. Content Reconstruction is the primary HDR reconstruction branch, while Exposure Restoration serves as auxiliary supervision. The two subnetworks share the same feature encoder but have different decoders. This symmetrical design allows the network to treat Exposure Restoration and Content Reconstruction as equally important tasks, and the separate two-stream design helps decouple the complexity of a single learning task.

[0102] Specifically, DERM first compresses the fused features using a convolutional block, reducing the number of channels from 3n. feat Reduced to n feat The encoded feature F is obtained. E :

[0103]

[0104] Next, f EThe data is simultaneously fed into two Residual Local Feature Networks (RLFNs), used for exposure restoration and content reconstruction, respectively. The RLFN uses a series of Residual Local Feature Blocks (RLFBs) (such as...) Figure 1 (As shown) feature enhancement. RLFB can effectively fuse local and global information, which is very beneficial for restoring texture details. The outputs of the two RLFNs are denoted as F. expo and F rec :

[0105] F expo =RLFN(F E );

[0106] F rec =RLFN(F E );

[0107] Where RLFN(·) represents the RLFN subnetwork. To further integrate the exposure recovery features F... expo and content reconstruction features F rec DERM uses another convolutional block to adaptively weight them:

[0108] F fused =Conv(F expo ,F rec ).

[0109] The decoder is used to optimize the recovery feature F fused Decode the image to obtain the HDR image prediction.

[0110] In practice, the decoder module includes decoder 1 and decoder 2.

[0111] Decoder 1 is used to convert F through a 3×3 convolutional layer and a sigmoid function. fused Mapping back to the original RGB space yields the final HDR prediction.

[0112]

[0113] Decoder 2 is used for F expo A 1×1 convolution and sigmoid activation were performed to obtain the exposure recovery prediction.

[0114] By setting up an exposure recovery branch as an auxiliary supervisor, the overexposed areas can be better recovered through joint supervision, while reconstructing rich texture details.

[0115] This setup performs exposure recovery and content reconstruction simultaneously through two symmetrical sub-networks, and uses convolutional blocks to adaptively fuse features from both branches. This two-stream structure and joint supervision enable better recovery of overexposed areas while reconstructing rich texture details, thus improving HDR image quality.

[0116] The entire HDR imaging process can be described as follows:

[0117]

[0118] In the formula, The network output is represented by [;,;], which represents the channel-wise chaining subroutine. f and ω represent the MPRNet and the network parameters to be learned. Input image Transform the image sequence into the HDR domain using the following formula:

[0119]

[0120] In the formula, t i L i The exposure time is γ, which is the gamma correction parameter, typically set to 2.2. Compared to the original LDR image, the HDR domain image contains more detail in the overexposed areas, making it easier to recover information from the saturated areas later.

[0121] S2. Train the constructed multi-scale progressive reconstruction network MPRNet.

[0122] In practice, this invention employs a joint reconstruction-exposure loss function for end-to-end training to obtain high-quality HDR reconstruction results.

[0123] Loss function of MPRNet, a multi-scale progressive reconstruction network for:

[0124]

[0125] Where α = 1 and β = 0.1 control the weights of exposure loss and perception loss, respectively; reconstruction loss Exposure loss and perceived loss These correspond to the content reconstruction branches of the multi-scale progressive reconstruction network. Exposure recovery branch And the perceptual quality of the reconstructed image.

[0126] Regarding reconstruction losses It employs a combination of L1 norm and μ-law tone mapping. The μ-law tone mapping function...

[0127]

[0128] Where μ is set to 5000. Using this mapping function, images in the HDR domain can be mapped to the LDR domain, making the supervision results more consistent with human perception. Reconstruction Loss The definition is:

[0129]

[0130] Where I H For Ground Truth HDR images.

[0131] Exposure recovery, as an auxiliary task, requires the design of a loss function to optimize exposure loss. This invention designs an improved Gaussian-weighted exposure loss function. Traditional exposure loss is calculated by predicting the image. and GT image I H While L2 norm in the brightness space is used to measure exposure recovery quality, this global averaging approach ignores local exposure differences. To better address the recovery of overexposed areas, this method introduces Gaussian weighting in the brightness space, assigning higher weights to the central region (close to 0.5). Furthermore, Gaussian filtering is used for post-processing to obtain a smooth weight map. (Exposure loss) The calculation process is as follows, where ε(x) is the function for calculating the exposure feature map:

[0132]

[0133]

[0134] Where w y = [0.299, 0.587, 0.114] represents the RGB to luminance (Y) conversion matrix. Let σ be a two-dimensional Gaussian kernel with standard deviation σ, where * denotes the convolution operation, and σ * Controlling the bandwidth of Gaussian weighting, ||A|| F Let Frobenius norm be the norm of matrix A.

[0135] To make the reconstructed image after tone mapping more consistent with human perception, a perceptual loss based on VGG16 was introduced.

[0136]

[0137] Where φ i (·) represents the output of the i-th feature block of VGG16, including layers 1-4, 5-9, 10-16, and 17-23. λi The weights are assigned to the loss of each feature block. Using features from shallow to deep layers helps to simultaneously optimize the similarity of low-level texture and high-level semantics.

[0138] S3. Use the trained multi-scale progressive reconstruction network MPRNet for actual high dynamic range imaging processing.

[0139] This invention constructs a novel Multi-Scale Progressive Reconstruction Network (MPRNet) specifically designed for HDR imaging in dynamic scenes. MPRNet innovatively combines an encoder / decoder, a Multi-Scale Progressive Reconstruction Module (MSPRM), and a Dual-Stream Exposure Recovery Module (DERM). MSPRM utilizes a feature pyramid to progressively refine motion artifacts across multiple scales, and adjusts and corrects motion information at both the same and cross-scale levels through an attention mechanism and a scale selection module. This breaks away from conventional single-scale feature fusion approaches and effectively models long-distance dependencies. DERM employs a dual-stream structure to simultaneously handle exposure recovery and content reconstruction, guided by a joint loss function, accurately repairing overexposed areas while preserving image details.

[0140] Through this innovative network architecture and multi-task optimization strategy, MPRNet effectively suppresses ghosting and improves the detail fidelity of HDR images generated from dynamic scenes. Extensive experiments on standard HDR datasets demonstrate that MPRNet significantly outperforms existing mainstream models in both qualitative and quantitative metrics, especially showing remarkable advantages when handling large-scale motion and extreme exposure conditions.

[0141] This invention proposes a novel end-to-end trainable network, MPRNet, which can generate high-quality, ghost-free HDR images from unaligned multi-exposure inputs. Extensive validation demonstrates that MPRNet significantly outperforms mainstream HDR methods in image structure preservation, motion ghosting removal, and texture detail preservation. Furthermore, a multi-scale feature fusion technique is introduced into MSPRM to adaptively integrate motion and exposure information, significantly improving HDR imaging processing in dynamic scenes. Additionally, this invention designs a DERM dual-stream exposure restoration module, which finely repairs overexposed areas while maintaining image content integrity, using a novel joint loss function to enhance exposure restoration and detail preservation.

[0142] To better understand the effectiveness of the multi-scale progressive reconstruction network MPRNet constructed by this method, the following experimental verification is provided.

[0143] Experimental verification

[0144] Dataset and Implementation Details

[0145] The Kalantari et al. dataset is a widely used benchmark for dynamic scene HDR imaging, containing 74 training scenes and 15 test scenes. Each scene consists of three LDR images with exposure biases of {-2, 0, 2} or {-3, 0, 3} and a resolution of 1000×1500. The training and quantitative evaluation of the multi-scale progressive reconstruction network MPRNet in this invention were performed entirely on the training and test sets of the Kalantari et al. dataset. In addition, generalization validation was performed on the Sen et al. dataset, as this dataset does not contain Ground Truth (GT) data and can better reflect the model's generalization performance.

[0146] Training Details. To effectively train the network, the same data augmentation strategies, such as cropping, flipping, and rotating, are applied to the original input images, expanding the dataset to four times its original size. The proposed MPRNet is implemented on the PyTorch framework and trained using an NVIDIA RTX 4090 GPU. It employs an Adam optimizer with an initial learning rate of 2e-4, β1 set to 0.9, β2 set to 0.999, ε set to 1e-8, and uses an ExponentialLR learning rate scheduler with a learning rate decay coefficient γ set to 0.93. To avoid overfitting, early stopping is employed during model training; training is stopped if the loss does not show significant improvement after five consecutive epochs.

[0147] Evaluation Metrics. Similar to mainstream evaluation methods, this experiment uses PSNR and SSIM to evaluate model performance. To further investigate the model's performance under different visual conditions, PSNR-l, PSNR-μ, SSIM-l, and SSIM-μ metrics were calculated between the generated HDR image and the ground truth (GT). "-μ" indicates that the metric was calculated in the tone mapping domain, and "-l" indicates that the metric was calculated in the linear domain. In addition, the HDR-VDP-2 metric was also calculated, which is a metric that measures human visual perception in the HDR domain.

[0148] Comparison with mainstream methods

[0149] Quantitative Results. Table 1 shows the quantitative comparison results of the proposed MPRNet with other representative methods on Kalantariet al.'s dataset. It can be seen that our method achieves the best performance in four metrics: PSNR-μ, SSIM-μ, SSIM-1, and HDR-VDP-2, second only to CA-ViT in PSNR-1. Compared to the second-place CA-ViT, our method has a 0.16 dB higher PSNR-μ value, the same SSIM-μ value, a 0.0003 higher SSIM-1 value, and a 0.08 higher HDR-VDP-2 value. This indicates that MPRNet can generate higher quality and more visually realistic HDR images. Compared with the classic AHDR method, our method improves PSNR-μ by 0.86 dB and HDR-VDP-2 by 1.50 dB. In the comparison of inference time, the Kalantari method consumed the longest time because the model requires optical flow alignment on the CPU in advance. The diffusion model method DiffHDR has a slower running speed due to its slow sampling diffusion process. However, the model of this invention has a significant advantage in inference performance thanks to its efficient network design.

[0150] Table 1

[0151]

[0152] Quantitative comparisons of this method with other mainstream methods on the Kalantari et al. dataset are presented, using PSNR, SSIM, and HDR-VDP-2 as evaluation metrics. "-μ" indicates the metric is calculated in the tone mapping domain, "-l" indicates it is calculated in the linear domain, and "Times" represents the model inference time. All values ​​are averaged results after evaluation across 15 test scenarios. The bolded portions in the table represent the highest values ​​for each metric.

[0153] Qualitative results. Figure 3This paper presents a visual comparison of our proposed method with several representative mainstream methods on the Kalantari et al. dataset. For fairness, the results of all methods were compared using the same tone mapping in Photomatix Pro software. It can be seen that the traditional Kalantari method exhibits significant tearing, artifacts, and color distortion. CNN-based AHDR and RNN-based HDRRNN show saturation and artifacts in large-scale motion regions. The generative method HDR-GAN introduces additional saturation and distortion due to generation, and suffers from pixel loss at misaligned edges. The Transformer method CA-ViT struggles to handle high-resolution blocks, resulting in exposure saturation and edge loss. In contrast, our proposed MPRNet effectively repairs overexposed areas and handles large-scale motion well. The generated HDR results outperform other compared methods in terms of ghosting suppression, texture preservation, and fidelity of misaligned edge information.

[0154] The performance differences can be attributed to the varying abilities of the methods in modeling and fusing motion and exposure information. Traditional methods struggle to handle large-scale motion and overexposed areas, while CNN-based methods have limited receptive fields. Transformer-based methods can model global information but face limitations when processing high-resolution blocks.

[0155] MPRNet's superior performance stems from its multi-scale progressive fusion and two-stream exposure reconstruction. It adaptively fuses motion information across scales and employs an attention mechanism to weight features at different exposures. The two-stream exposure reconstruction module decouples exposure restoration and content reconstruction, guiding the repair of overexposed areas. This enables MPRNet to effectively suppress ghosting, restore details, and achieve better results in objective metrics and visual quality.

[0156] To verify the generalization performance of this method, it was tested on the Sen et al. dataset ( Figure 4 (a) and Figure 4 Further testing was conducted on (b) of the dataset. This dataset does not contain Ground Truth HDR images, so only qualitative comparisons could be performed. Figure 4 This paper presents the results of our proposed MPRNet on two scenes compared to Kalantari, AHDR, and HDR-GAN on the Sen et al. dataset. It can be seen that even on datasets without Ground Truth, our method can still generate high-quality HDR images with no ghosting and good exposure, demonstrating superior generalization ability compared to other methods. This indicates that by adaptively fusing features in the feature space, MPRNet possesses a certain degree of scene generalization and motion adaptation capabilities. Of course, truly achieving generalization to any scene remains an open challenge.

[0157] ablation experiment

[0158] This experiment conducted an ablation study on the Kalantari et al. dataset to verify the progressiveness of MSPRM (including scale progressiveness and fusion stage progressiveness) and the impact of the DERM module on MPRNet performance. As shown in Table 2, four variants were designed. Quantitative results from the ablation experiments are presented.

[0159] Table 2 Quantitative results of the ablation experiment.

[0160]

[0161] In the table, "-s" or "-m" represent single or multiple scales and fusion stages, respectively.

[0162] Quantitative results in the table and Figure 5 Qualitative comparisons demonstrate the effectiveness and necessity of the proposed MSPRM and DERM modules in achieving superior HDR reconstruction performance. The base model MPRNet-B suffers from severe ghosting artifacts and edge distortion. MPRNet-S employs multi-scale fusion to mitigate edge distortion and improve quantization metrics, but ghosting artifacts still persist due to significant motion. MPRNet-M, through multi-stage and multi-scale fusion, effectively suppresses motion-induced ghosting and enhances the visual quality of misaligned edges, resulting in a significant improvement in PSNR-1. However, it still lacks detail in overexposed areas (such as...). Figure 5 (As shown). The complete MPRNet model, by integrating the DERM module, achieves optimal results across all metrics by restoring details in overexposed areas and optimizing the reconstructed HDR image in terms of exposure and detail fidelity.

[0163] In summary, the ablation experiments highlight the importance of MSPRM for progressive, multi-stage, multi-scale fusion when handling complex motion and scenes, and the crucial role of DERM in recovering overexposed areas. The combination of these modules enables MPRNet to generate high-quality HDR images with exceptional detail fidelity and ghosting suppression.

[0164] in conclusion

[0165] This invention proposes a novel multi-scale progressive reconstruction network (MPRNet) for high-quality HDR imaging in dynamic scenes. Addressing the ghosting, blurring, and distortion issues encountered by existing methods in large-scale motion and extreme exposure scenarios, MPRNet cleverly incorporates a multi-scale progressive reconstruction module (MSPRM) and a dual-stream exposure recovery module (DERM).

[0166] MSPRM introduces a context embedding module to model global dependencies at each scale of the feature pyramid. It integrates motion information from the same and higher levels into the current scale through a scale selection mechanism, and then uses an attention mechanism to achieve adaptive weighting of different exposure features within the same scale, thereby maximizing the use of complementary information from multi-scale and multi-exposure images.

[0167] DERM employs a symmetrical two-stream structure to simultaneously perform exposure restoration and content reconstruction, and guides the fine-grained repair of overexposed areas through a joint loss function.

[0168] Experiments on several challenging HDR datasets demonstrate that our proposed MPRNet method can generate high-quality HDR images with no ghosting, proper exposure, and rich detail, significantly outperforming existing methods. Ablation studies further validate the effectiveness of multi-scale progressive fusion and dual-stream exposure restoration. Furthermore, its good performance in scenes without Ground Truth confirms MPRNet's generalization ability.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A high dynamic range imaging method based on a multi-scale progressive reconstruction network, characterized in that, Includes the following steps: S1. Construct the multi-scale progressive reconstruction network MPRNet, including an encoder module, a multi-scale progressive reconstruction module MSPRM, a dual-stream exposure recovery module DERM, and a decoder module. The encoder module is used to extract features from the three LDR images with different exposure levels, respectively, to obtain the corresponding feature maps E1, E2, and E3. The Multi-Scale Progressive Reconstruction Module (MSPRM) is used to perform multi-scale progressive reconstruction and global context modeling on feature maps E1, E2, and E3 to obtain fused features. The dual-stream exposure recovery module (DERM) is used for fusion features. Exposure recovery of the overexposed area was performed, and the exposure recovery feature F was obtained. expo It is also used for fusion features. The content details are supplemented and repaired to obtain the content reconstruction feature F. rec It is also used for the exposure recovery feature F. expo and content reconstruction features F rec Adaptive weighting is performed to obtain the optimized recovery feature F. fused ; The decoder is used to optimize the recovery feature F fused Decode the image to obtain the HDR image prediction. S2. Train the constructed multi-scale progressive reconstruction network MPRNet; S3. Use the trained multi-scale progressive reconstruction network MPRNet for actual high dynamic range imaging processing.

2. The high dynamic range imaging method based on multi-scale progressive reconstruction networks as described in claim 1, characterized in that: The workflow of the multi-scale progressive refactoring module MSPRM includes: First, a feature pyramid with three scales is created through downsampling. Next, for the i-th feature at the s-th scale in the t-th stage... CE module embedded in context Perform global context modeling and output enhanced features. Where NL(·) represents context modeling, which aggregates global context information by calculating the correlation between any two locations; Then, the scale selection module (SSM) fuses features from the same and higher levels into the current scale, resulting in cross-scale fused features. Then, a separable large kernel attention mechanism is used to fuse features at the same level to obtain the feature representation for the next stage. Then, The finely aligned output features are obtained by sequentially passing through a CE module, an SSM module, and a convolutional attention block CA. Finally, the output features at the original scale are... It is passed to the dual-stream exposure recovery module DERM.

3. The high dynamic range imaging method based on multi-scale progressive reconstruction networks as described in claim 2, characterized in that: The input of SSM includes an upsampling branch and a fusion branch; Upsampling branch will extract high-level features Mapping to the s-th scale to obtain features Merging branches will combine sibling features The outputs of the two branches are passed to the output; the outputs of the two branches are fused through the SSM module to obtain the cross-scale fused features.

4. The high dynamic range imaging method based on multi-scale progressive reconstruction networks as described in claim 3, characterized in that: When the SSM module fuses the outputs of the two branches, it obtains the feature map through global average pooling. The global response is obtained by performing a 1×1 convolution and softmax to obtain nonlinear mapping weight maps at two different scales. in, This represents element-wise multiplication.

5. The high dynamic range imaging method based on multi-scale progressive reconstruction networks as described in claim 1, characterized in that: DERM includes an exposure recovery subnetwork for exposure recovery and a content reconstruction subnetwork for content reconstruction; the exposure recovery subnetwork and the content reconstruction subnetwork have the same structure and share the same feature encoder.

6. The high dynamic range imaging method based on multi-scale progressive reconstruction networks as described in claim 5, characterized in that: The working process of the dual-stream exposure recovery module (DERM) includes: First, the features are fused using a convolutional block. Compression was performed, reducing the number of channels from 3n feat Reduced to n feat The encoded feature F is obtained. E : After that, f E Simultaneously, the data is fed into the exposure recovery subnetwork and the content reconstruction subnetwork, respectively, for exposure recovery and content reconstruction, to obtain the exposure recovery feature F. expo and content reconstruction features F rec Among them, both exposure recovery from the network and content reconstruction from the network are RLFN networks; F expo =RLFN(F E ); F rec =RLFN(F E ); Wherein, RLFN(·) represents the RLFN subnetwork; Then, another convolutional block is used to refine the exposure recovery feature F. expo and content reconstruction features F rec Adaptive weighting is performed to obtain the optimized recovery feature F. fused : F fused =Conv(F expo ,F rec )。 7. The high dynamic range imaging method based on multi-scale progressive reconstruction networks as described in claim 1, characterized in that: The decoder module includes decoder 1 and decoder 2; Decoder 1 is used to convert F through a 3×3 convolutional layer and a sigmoid function. fused Mapping back to the original RGB space yields the final HDR prediction. Decoder 2 is used for F expo A 1×1 convolution and sigmoid activation were performed to obtain the exposure recovery prediction.

8. The high dynamic range imaging method based on multi-scale progressive reconstruction networks as described in claim 7, characterized in that: Loss function of MPRNet, a multi-scale progressive reconstruction network for: Where α and β control the weights of exposure loss and perception loss, respectively; reconstruction loss Exposure loss and perceived loss These correspond to the content reconstruction branches of the multi-scale progressive reconstruction network. Exposure recovery branch And the perceptual quality of the reconstructed image.

9. The high dynamic range imaging method based on multi-scale progressive reconstruction networks as described in claim 8, characterized in that: Reconstruction losses for: Among them, I H For Ground Truth HDR images; μ is a preset fixed value; Exposure loss for: Where ε(x) is the exposure feature map calculation function; w y = [0.299, 0.587, 0.114], representing the RGB to luminance (Y) conversion matrix; Let σ be a two-dimensional Gaussian kernel with standard deviation σ, where * denotes the convolution operation, and σ * Controlling the bandwidth of Gaussian weighting, ||A|| F Denotes the Frobenius norm of matrix A; Perceived loss The loss function based on VGG16 is as follows: Where, φ i (·) represents the output of the i-th feature block of VGG16; λ i The weights are the loss values ​​for each feature block.

Citation Information

Patent Citations

  • High dynamic range multi-exposure image fusion model and method based on attention mechanism

    CN114331931A

  • Ghosting-free high-dynamic-range light field imaging method for dynamic scene

    CN117237207A