Ultrahigh dynamic range imaging system and method based on diffusion model image prior

Abstract

The invention relates to an ultrahigh dynamic range imaging system and method based on diffusion model image priori, the system comprises a pre-alignment module, a variational auto-encoder, a decomposition fusion control module, a fidelity control module and a variational auto-decoder, and the method comprises the following steps: converting and aligning a short exposure image to a long exposure image by adopting an optical flow estimation algorithm; covering the shielding area to obtain a pre-alignment result; performing information decomposition on the short-exposure image and the long-exposure image, and performing fusion and tone mapping on the pre-alignment result by controlling the fusion process of a diffusion model to obtain a fusion result; and according to the long exposure image and the pre-alignment result, carrying out fidelity control on the fusion result, and decoding to obtain a reconstructed image. Compared with the prior art, the method has the advantages that the alignment and fusion robustness under the condition of large exposure level difference can be improved, and a natural and vivid tone mapping effect is generated in an extremely high dynamic range scene.

Description

Technical Field

[0001] The present invention relates to the technical field of high dynamic range imaging, and in particular to an ultra-high dynamic range imaging system and method based on diffusion model image prior. Background Art

[0002] Capturing high-dynamic range (HDR) scenes is one of the key challenges in camera system design. Currently, most cameras use exposure fusion technology to extend the dynamic range by fusing images taken at different exposure levels.

[0003] High dynamic range (HDR) imaging can be divided into two methods according to the domain in which the fusion occurs. One is the HDR fusion method, which first fuses multiple images of different exposures to generate an HDR image, and then compresses it into a low dynamic range (LDR) image through tone mapping for visualization or display; the other is the multi-exposure fusion method, which directly fuses multiple images of different exposures and outputs a low dynamic range image, eliminating the tone mapping step.

[0004] In the field of HDR fusion, early Kalantari et al. [1] proposed an HDR fusion method based on alignment and fusion, laying an important foundation for this field. Subsequently, researchers continued to improve the alignment process by developing more advanced modules to solve the artifact problem caused by motion between different exposures. Steven Tel et al. [2] Furthermore, a HDR fusion method without explicit alignment is proposed, which significantly simplifies the alignment step. Recently, Kong et al. [3] A novel and efficient HDR fusion technique based on optical flow is proposed. In addition, some studies [4,5] Attempts to implement HDR fusion algorithms based on the diffusion model framework have not yet fully utilized the powerful prior knowledge contained in the pre-trained diffusion model. Although the above methods have made some progress in the field of HDR fusion, they are generally only applicable to scenes with small exposure differences (usually 3-4 stops). When applied to ultra-high dynamic range scenes (i.e., scenes with large exposure differences), these methods often have difficulty in generating high-quality fusion results due to alignment errors of the input images, inconsistent lighting conditions, or artifacts in the tone mapping process.

[0005] In the field of multi-exposure fusion, existing research [6,7,8] Mainly for static scenes, multiple images are directly fused to generate low dynamic range images through supervised or self-supervised methods. This method is usually difficult to generate high-quality fusion results when dealing with complex scenes with motion scenes or large exposure differences.

[0006] In summary, in order to process dynamic scenes, most HDR fusion algorithms in the prior art usually align the input frames first. When there is a large brightness difference between the input frames, alignment becomes extremely difficult, which often leads to the appearance of "ghosting" problems.

[0007] In addition, most HDR algorithms assume that an underexposed image is just a darker version of a normally exposed image. However, the appearance of an object may change significantly as the exposure level changes, a phenomenon that often leads to unnatural fusion results and affects image quality.

[0008] The result of existing technology fusion is usually an HDR image, but because ordinary low dynamic range display devices cannot directly present HDR images, these images need to be further compressed through tone mapping. When the dynamic range is too high, tone mapping may introduce additional problems, such as loss of natural contrast and details, thus affecting the final output effect.

[0009] Prior art references:

[0010] [1]Kalantari NK,Ramamoorthi R.Deep high dynamic range imaging ofdynamic scenes[J].ACM Trans.Graph.,2017,36(4):144:1-144:12.

[0011] [2]Tel S, Wu Z, Zhang Y, et al.Alignment-free HDR Deghosting withSemantics Consistent Transformer[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision.2023:12836-12845.

[0012] [3]Kong L, Li B, Xiong Y, et al.SAFNet: Selective Alignment FusionNetwork for Efficient HDR Imaging[C] / / European Conference on ComputerVision.Springer,Cham,2025:256-273.

[0013] [4]Yan Q,Hu T,Sun Y,et al.Towards high-quality hdr deghosting withconditional diffusion models[J].IEEE Transactions on Circuits and Systems forVideo Technology,2023.

[0014] [5]Hu T,Yan Q,Qi Y,et al.Generating content for hdr deghosting fromfrequency view[C] / / Proceedings of the IEEE / CVF Conference on Computer Visionand Pattern Recognition.2024:25732-25741.

[0015] [6]Liang P,Jiang J,Liu X,et al.Fusion from decomposition:Aself-supervised decomposition approach for image fusion[C] / / European Conference onComputer Vision.Cham:Springer Nature Switzerland,2022:719-735.

[0016] [7]Jiang T,Wang C,Li X,et al.Meflut:Unsupervised 1d lookup tables formulti-exposure image fusion[C] / / Proceedings of the IEEE / CVF InternationalConference on Computer Vision.2023:10542-10551.

[0017] [8]Wu G, Fu H, Liu J, et al.Hybrid-supervised dual-search: Leveraging automatic learning for loss-free multi-exposure image fusion[C] / / Proceedings of the AAAI Conference on Artificial Intelligence.2024,38(6):5985-5993. Summary of the invention

[0018] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an ultra-high dynamic range imaging system and method based on diffusion model image prior, which can improve the alignment and fusion robustness under conditions of large differences in exposure levels and generate natural and realistic tone mapping effects in extremely high dynamic range scenes.

[0019] The object of the present invention can be achieved by the following technical scheme: an ultra-high dynamic range imaging system based on diffusion model image prior, comprising a pre-alignment module, a variational autoencoder, a decomposition and fusion control module, a fidelity control module and a variational autodecoder, wherein the pre-alignment module is used to align a short exposure image to a long exposure image to generate a pre-alignment result;

[0020] The variational autoencoder is used to encode the long exposure image to obtain an encoding result;

[0021] The decomposition and fusion control module performs information fusion and tone mapping on the input exposure image based on the pre-alignment result and the encoding result, and obtains a fusion result by controlling the fusion process of the diffusion model;

[0022] The fidelity control module is used to control the fidelity of the fusion result based on the long exposure image and the pre-alignment result;

[0023] The variational self-decoder is used to decode the fusion result to obtain a reconstructed image.

[0024] Furthermore, the pre-alignment module uses an optical flow estimation algorithm to align the short exposure image to the long exposure image and masks the occluded area to generate a pre-alignment result.

[0025] Further, the decomposition and fusion control module includes a short-burst feature extraction unit, a long-burst feature extraction unit and a fusion unit, wherein the short-burst feature extraction unit is used to extract image features of the short-exposure image;

[0026] The long-exposure feature extraction unit is used to extract the image features of the long-exposure image from the encoding result;

[0027] The fusion unit is based on a diffusion model and utilizes prior knowledge to fuse and tone map the image features of the short exposure image and the long exposure image.

[0028] Further, the short-burst feature extraction unit includes a plurality of cascaded short-burst feature extractors;

[0029] The long burst feature extraction unit includes a plurality of cascaded long burst feature extractors.

[0030] Furthermore, the fusion unit adopts a cross-attention mechanism and a multi-scale cross-attention mechanism to fuse the image features of the short exposure image and the long exposure image.

[0031] An ultra-high dynamic range imaging method based on diffusion model image prior comprises the following steps:

[0032] S1. Use the optical flow estimation algorithm to transform and align the short exposure image to the long exposure image, and mask the occluded area to obtain the pre-alignment result;

[0033] S2, performing information decomposition on the short exposure image and the long exposure image, and fusing and tone mapping the pre-alignment results by controlling the fusion process of the diffusion model to obtain a fusion result;

[0034] S3. According to the long exposure image and the pre-alignment result, the fidelity of the fusion result is controlled, and then the reconstructed image is obtained by decoding.

[0035] Furthermore, the step S1 includes the following steps:

[0036] S11, using an optical flow estimation algorithm to process the long exposure image and the short exposure image respectively to obtain corresponding first optical flows and second optical flows;

[0037] S12, performing consistency check on the first optical flow and the second optical flow to obtain an occlusion mask;

[0038] S13: aligning the short exposure image and the first optical flow, and masking the occluded area using an occlusion mask to generate a pre-alignment result.

[0039] Furthermore, the step S2 comprises the following steps:

[0040] S21, performing encoding processing on the long exposure image to obtain an encoding result;

[0041] S22, performing short burst feature extraction and long burst feature extraction on the short exposure image and the encoding result respectively;

[0042] S23. Use the diffusion model to fuse the short burst features and the long burst features, complete the fusion and tone mapping of the pre-alignment results, and obtain the fusion result.

[0043] Furthermore, the specific process of step S22 is as follows:

[0044] For short exposure images, the brightness component of the short exposure images is removed, the structure and color information is extracted, and the short burst features of the short exposure images are extracted through several layers of convolution using the same network structure as ControlNet;

[0045] For long exposure images, long burst features are extracted from the latent variables of the encoding results.

[0046] Furthermore, the step S23 specifically combines the cross-attention mechanism and the multi-scale cross-attention mechanism for fusion processing.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] The present invention proposes an ultra-high dynamic range imaging system based on diffusion model image prior, including a pre-alignment module, a variational autoencoder, a decomposition and fusion control module, a fidelity control module and a variational autodecoder. The pre-alignment module is used to align a short exposure image to a long exposure image to generate a pre-alignment result; the variational autoencoder is used to encode the long exposure image to obtain an encoding result; the decomposition and fusion control module is used to perform information fusion and tone mapping on the input exposure image based on the pre-alignment result and the encoding result by controlling the fusion process of the diffusion model to obtain a fusion result; and the fidelity control module is used to control the fidelity of the fusion result based on the long exposure image and the pre-alignment result. In this way, the alignment and fusion problems with large differences in exposure levels can be solved, and the tone mapping can be adaptively learned through the image prior of the diffusion model, thereby solving the tone mapping problem of ultra-high dynamic range, and the fidelity of the reconstructed image can be effectively maintained.

[0049] The present invention uses an optical flow estimation algorithm to transform and align the short exposure image to the long exposure image, and masks the occluded area to obtain a pre-aligned result, thereby having better robustness in dealing with potential alignment errors and illumination changes.

[0050] The present invention designs a decomposition and fusion control scheme, which removes the brightness component of the short exposure image, extracts its structure and color information, and combines the cross-attention mechanism and the multi-scale cross-attention mechanism to enhance the feature fusion with the long exposure image. At the same time, the prior information of the diffusion model is utilized to improve the robustness of the fusion.

[0051] The present invention designs an additional fidelity control scheme, which inputs the long exposure image and the pre-alignment result into the fidelity control module to ensure the fidelity of the fusion result, and controls the fusion result to be highly consistent with the input texture. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0053] Figure 2 This is a schematic diagram of the pre-alignment process in the embodiment;

[0054] Figure 3 It is a schematic diagram of the decomposition and fusion control process in the embodiment;

[0055] Figure 4 Schematic diagram of image fusion and tone mapping process in the embodiment;

[0056] Figure 5 and Figure 6 Schematic diagram for comparing imaging effects in the embodiments. DETAILED DESCRIPTION

[0057] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] Example

[0059] An ultra-high dynamic range imaging system based on diffusion model image prior includes a pre-alignment module, a variational autoencoder, a decomposition and fusion control module, a fidelity control module and a variational autodecoder. The pre-alignment module is used to align a short-exposure image to a long-exposure image to generate a pre-alignment result. Specifically, an optical flow estimation algorithm can be used to align the short-exposure image to the long-exposure image and mask the occluded area to generate a pre-alignment result.

[0060] The variational autoencoder is used to encode the long exposure image to obtain an encoding result;

[0061] The decomposition fusion control module performs information fusion and tone mapping on the input exposure image based on the pre-alignment result and the encoding result, and obtains the fusion result by controlling the fusion process of the diffusion model;

[0062] Specifically, the decomposition and fusion control module includes a short burst feature extraction unit, a long burst feature extraction unit and a fusion unit, wherein the short burst feature extraction unit is used to extract image features of the short exposure image, and multiple cascaded short burst feature extractors may be used;

[0063] The long burst feature extraction unit is used to extract the image features of the long exposure image from the encoding result, and can use multiple cascaded long burst feature extractors;

[0064] The fusion unit is based on the diffusion model, adopts the cross-attention mechanism and the multi-scale cross-attention mechanism, and uses the prior knowledge to fuse and tone map the image features of the short-exposure image and the long-exposure image.

[0065] The fidelity control module is used to control the fidelity of the fusion result based on the long exposure image and the pre-alignment result;

[0066] The variational self-decoder is used to decode the fusion result to obtain the reconstructed image.

[0067] Based on the above system, an ultra-high dynamic range imaging method based on diffusion model image prior is implemented, such as Figure 1 As shown, the following steps are included:

[0068] S1. Use the optical flow estimation algorithm to transform and align the short exposure image to the long exposure image, and mask the occluded area to obtain the pre-alignment result;

[0069] S2, performing information decomposition on the short exposure image and the long exposure image, and fusing and tone mapping the pre-alignment results by controlling the fusion process of the diffusion model to obtain a fusion result;

[0070] S3. According to the long exposure image and the pre-alignment result, the fidelity of the fusion result is controlled, and then the reconstructed image is obtained by decoding.

[0071] This embodiment applies the above solution to build a pre-alignment module, a diffusion model network, a decomposition and fusion control module, and a fidelity control module to achieve the following: Figure 2 The pre-alignment process shown in (using optical flow estimation to transform the short exposure image to the long exposure image, and masking the occluded area to obtain the pre-aligned result) is as follows Figure 3 The decomposition fusion control process shown in FIG. 1 (by removing the brightness component of the short exposure image, extracting its structure and color information, and combining the cross attention mechanism and the multi-scale cross attention mechanism to enhance the feature fusion with the long exposure image), as shown in FIG. Figure 4 The image fusion and tone mapping process shown (using the prior knowledge of the diffusion model, the results of pre-alignment of the long exposure image and the short exposure image are fused and tone mapped, and an additional fidelity control module is used to control the fidelity of the reconstruction result).

[0072] Specifically, the pre-alignment module is first used to align the short exposure image to the long exposure image to generate a pre-alignment result. Then, the long exposure image is encoded by the variational autoencoder to obtain the encoding result. Subsequently, the encoding result and the pre-aligned image are input into the decomposition and fusion control module, which is responsible for controlling the diffusion model network to fuse and tone map the different exposure images and finally generate the fusion result. Finally, the long exposure image and the pre-alignment result are input into the fidelity control module to ensure the fidelity of the fusion result, and finally the reconstructed image is obtained.

[0073] Among them, the pre-alignment module is used to align two images with different exposures and generate pre-alignment results, which provide a basis for subsequent information fusion. Figure 2 As shown, the optical flow estimation algorithm RAFT is used to align the short exposure image to the long exposure image, and the occluded area is masked to generate a pre-alignment result.

[0074] The decomposition and fusion control module is used to decompose the information of the input images with different exposures and control the fusion process of the diffusion model, thereby effectively realizing information fusion and tone mapping. Figure 4 As shown in the figure, this module removes the brightness component of the short-exposure image, extracts its structure and color information, and uses the same network structure as ControlNet to extract the information of the short-exposure image through several layers of convolution, and then fuses it with the long-exposure image features using the cross-attention mechanism. In addition, the module also uses a multi-scale cross-attention mechanism to enhance the feature fusion with the long-exposure image and retain more details when generating guidance information to ensure the accuracy of the fusion result.

[0075] The diffusion model network is used to fuse and tone map images with different exposures. Figure 3 As shown in FIG. 1 , the module uses the pre-trained diffusion model Stable Diffusion to fuse the pre-aligned results of the long exposure image and the short exposure image with its prior knowledge and realize tone mapping.

[0076] The fidelity control module is used to ensure the fidelity of the fusion results. Figure 3 As shown in Figure 1, this module controls the fidelity of the reconstructed image to ensure that the fusion result is highly consistent with the input texture. The fidelity control module uses the same network structure as the decomposition and fusion control module to further improve the authenticity of the fusion effect.

[0077] To verify the effectiveness of this solution, this embodiment uses this solution and the existing HDR fusion method and multi-exposure fusion method to process different exposure images respectively and compare the imaging results. Figure 5 and Figure 6As shown in the figure, it can be seen that this scheme is highly robust to potential alignment errors or illumination changes, and can generate natural tone mapping effects in scenes with very high dynamic ranges. Even in complex scenes with exposure differences of up to 9 stops (-6EV to +3EV), it can still generate beautiful and high-quality imaging results.

[0078] In summary, this scheme proposes a fusion and tone mapping imaging scheme suitable for ultra-high dynamic range dynamic scenes based on the image prior of the diffusion model:

[0079] 1. To solve the alignment problem with large differences in exposure levels, the short-exposure image is transformed and aligned to the long-exposure image, and the occluded area is masked.

[0080] 2. In order to solve the fusion problem of large differences in exposure levels, a new decomposition-fusion control branch is proposed. By removing the brightness component of the short-exposure image, its structure and color information are extracted, and the cross-attention mechanism and multi-scale cross-attention mechanism are used to enhance the feature fusion with the long-exposure image. At the same time, the prior information of the diffusion model is used to improve the robustness of the fusion.

[0081] 3. To solve the ultra-high dynamic range tone mapping problem, the image prior of the diffusion model is used to adaptively learn the tone mapping.

[0082] 4. In order to ensure that the generated output is highly consistent with the real scene, an additional fidelity control branch is proposed to effectively maintain the fidelity of the resulting image.

[0083] Compared with traditional HDR fusion technology, this solution has better fusion effect. It follows the multi-exposure fusion process, and the exposure fusion directly generates LDR output, avoiding cascade errors.

[0084] This scheme is more robust to potential alignment errors and illumination changes. The short exposure image is used as a soft guide rather than a hard constraint, so it is more robust to alignment errors and illumination changes.

[0085] The image output by this scheme has better tone results. The image prior of the generated model can ensure the natural appearance of the output image, thereby reducing the generation of potential artifacts and generating natural and realistic tone mapping effects in extremely high dynamic range scenes.

Claims

1. An ultra-high dynamic range imaging system based on diffusion model image prior, characterized in that: It includes a pre-alignment module, a variational autoencoder, a decomposition and fusion control module, a fidelity control module and a variational autodecoder, wherein the pre-alignment module is used to align the short-exposure image to the long-exposure image to generate a pre-alignment result; The variational autoencoder is used to encode the long exposure image to obtain an encoding result; The decomposition and fusion control module performs information fusion and tone mapping on the input exposure image based on the pre-alignment result and the encoding result, and obtains a fusion result by controlling the fusion process of the diffusion model; The fidelity control module is used to control the fidelity of the fusion result based on the long exposure image and the pre-alignment result; The variational self-decoder is used to decode the fusion result to obtain a reconstructed image.

2. The ultra-high dynamic range imaging system based on diffusion model image prior according to claim 1, characterized in that: The pre-alignment module uses an optical flow estimation algorithm to align the short exposure image to the long exposure image and mask the occluded area to generate a pre-alignment result.

3. The ultra-high dynamic range imaging system based on diffusion model image prior according to claim 1, characterized in that: The decomposition and fusion control module includes a short burst feature extraction unit, a long burst feature extraction unit and a fusion unit, wherein the short burst feature extraction unit is used to extract image features of the short exposure image; The long-exposure feature extraction unit is used to extract the image features of the long-exposure image from the encoding result; The fusion unit is based on a diffusion model and utilizes prior knowledge to fuse and tone map the image features of the short exposure image and the long exposure image.

4. The ultra-high dynamic range imaging system based on diffusion model image prior according to claim 3, characterized in that: The short burst feature extraction unit includes a plurality of cascaded short burst feature extractors; The long burst feature extraction unit includes a plurality of cascaded long burst feature extractors.

5. The ultra-high dynamic range imaging system based on diffusion model image prior according to claim 3, characterized in that: The fusion unit adopts a cross-attention mechanism and a multi-scale cross-attention mechanism to fuse the image features of the short exposure image and the long exposure image.

6. An ultra-high dynamic range imaging method based on diffusion model image prior, characterized in that: The following steps are involved: S1. Use the optical flow estimation algorithm to transform and align the short exposure image to the long exposure image, and mask the occluded area to obtain the pre-alignment result; S2, performing information decomposition on the short exposure image and the long exposure image, and fusing and tone mapping the pre-alignment results through the fusion process of the control diffusion model to obtain a fusion result; S3. According to the long exposure image and the pre-alignment result, the fidelity of the fusion result is controlled, and then the reconstructed image is obtained by decoding.

7. The ultra-high dynamic range imaging method based on diffusion model image prior according to claim 6, characterized in that: The step S1 comprises the following steps: S11, using an optical flow estimation algorithm to process the long exposure image and the short exposure image respectively to obtain corresponding first optical flows and second optical flows; S12, performing consistency check on the first optical flow and the second optical flow to obtain an occlusion mask; S13: aligning the short exposure image and the first optical flow, and masking the occluded area using an occlusion mask to generate a pre-alignment result.

8. The ultra-high dynamic range imaging method based on diffusion model image prior according to claim 6, characterized in that: The step S2 comprises the following steps: S21, performing encoding processing on the long exposure image to obtain an encoding result; S22, performing short burst feature extraction and long burst feature extraction on the short exposure image and the encoding result respectively; S23. Use the diffusion model to fuse the short burst features and the long burst features, complete the fusion and tone mapping of the pre-alignment results, and obtain the fusion result.

9. The ultra-high dynamic range imaging method based on diffusion model image prior according to claim 8, characterized in that: The specific process of step S22 is as follows: For short exposure images, the brightness component of the short exposure images is removed, the structure and color information is extracted, and the short burst features of the short exposure images are extracted through several layers of convolution using the same network structure as ControlNet; For long exposure images, long burst features are extracted from the latent variables of the encoding results.

10. The ultra-high dynamic range imaging method based on diffusion model image prior according to claim 8, characterized in that: The step S23 specifically combines the cross-attention mechanism and the multi-scale cross-attention mechanism for fusion processing.

Images