High dynamic range image tone mapping method based on diffusion model and zero sample training

By integrating the tone mapping network model in the diffusion model, using structural information and brightness distribution maps, the problem of poor tone mapping effect in the existing technology is solved, and efficient tone mapping under extreme lighting conditions is achieved, and the original color is retained Graphic information and improve the tone nature.

CN119006347AActive Publication Date: 2024-11-22FUDAN UNIVERSITY
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411020264.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-22
Estimated Expiration
2044-07-29

AI Technical Summary

Technical Problem

The existing high dynamic range image tone mapping methods cannot effectively retain the original image information, and at the same time the effect is poor under extreme lighting conditions, resulting in the failure to retain the original image information well and lack of natural tones.

Method used

Using a diffusion model and zero-sample training method, a tone mapping network model is constructed, integrated into the diffusion model framework, using the structural information and rough brightness distribution map of the original image as conditional input, correcting the reverse sampling results of the diffusion model, and combining the structure Information and brightness distribution maps to enhance the details of the output image.

Benefits of technology

The effect of high dynamic range image tone mapping under extreme lighting conditions is realized, which can retain the original image information well and improve the tone mapping effect. It only requires training the network with low dynamic range images to be used for tone mapping of high dynamic range images. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006347A_ABST
    Figure CN119006347A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a high dynamic range image tone mapping method based on a diffusion model and zero sample training. The method comprises the following steps of: constructing a tone mapping network model, integrating the tone mapping network model into a framework of a diffusion model, taking structure information and a rough brightness distribution diagram as condition input of the diffusion model, and correcting an intermediate result of reverse sampling of the diffusion model by utilizing the structure information of an original diagram. The network model comprises a generation model and a control module thereof; the generation model comprises cascaded four-scale coding modules, a middle module and cascaded four-scale decoding modules; the control module comprises four-scale coding modules which are the same as the SD model, and a zero convolution layer is finally added to the network of each scale. And the output of each scale of the control module is added to the feature map of the output of the corresponding scale of the generative model so as to control the final output result of the generative model. According to the method, the original image information can be well reserved, and the tone mapping effect of the HDR image is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a high dynamic range image tone mapping method based on a diffusion model and zero-sample training. Background Art

[0002] High Dynamic Range (HDR) images have a larger bit width than ordinary images, so they can record richer information, especially under extreme lighting conditions. However, HDR images cannot be displayed directly on ordinary display devices, so tone mapping is required to map the pixel values ​​of the HDR image to the displayable Low Dynamic Range (LDR), while generating normal tones and retaining the rich information recorded in the original image as much as possible.

[0003] However, it is theoretically impossible to obtain a paired HDR-high-quality LDR dataset. Existing advanced methods [1][2] mostly use generative adversarial networks to solve this problem, using a loss function that includes a structure preservation term and a style naturalness term. However, the optimization directions of these two objectives may be different, resulting in the result being unable to well preserve the original image information and lacking natural tones. There are also some works [3] that train models on datasets for low-light image enhancement, but the results are not good for HDR image tone mapping tasks. Summary of the invention

[0004] The purpose of the present invention is to provide a high dynamic range image tone mapping method based on a diffusion model and zero-sample training, so as to well preserve the original image information and improve the HDR image tone mapping effect.

[0005] The high dynamic range image tone mapping method based on diffusion model and zero-sample training provided by the present invention includes constructing a tone mapping network model and integrating it into the framework of the diffusion model, taking structural information and a rough brightness distribution map as conditional inputs of the diffusion model, and using the structural information of the original image to correct the intermediate result of the reverse sampling of the diffusion model. Figure 1 , the specific steps are:

[0006] (1) Construct a tone mapping network model for each step of the diffusion model. The network model structure includes a generation model for pre-training and a control module of the model, such as Figure 2As shown. The pre-trained generative model adopts the network structure of the stable diffusion model [4] (Stable Diffusion, hereinafter referred to as the SD model), which mainly includes: cascaded encoding modules of four scales, an intermediate module, and cascaded decoding modules of four scales; the structure of the control module includes the same four-scale encoding modules as the SD model, but a zero convolution layer is added at the end of the neural network of each scale. The output of each scale of the control module is added to the feature map of the corresponding scale output of the generative model to control the final output result of the generative model. The input of the control module includes the structural information and the approximate brightness distribution map at each pixel of the image. Among them:

[0007] The calculation formula for the structural information at each pixel is:

[0008]

[0009] In the formula, I ori represents the original image, μ ori and σ ori They represent the local Gaussian mean and local standard deviation of the image respectively, ε is a small value introduced to prevent division by 0, (i, j) is the coordinate of the current pixel in the image; μ ori and σ ori The specific calculation formula is:

[0010]

[0011] Among them, K represents the window size of Gaussian filtering, and ω represents the Gaussian filtering kernel.

[0012] The calculation method of the brightness distribution map is different in the training stage and the test stage. In the training stage, the calculation method is to perform mean filtering on the brightness channel of the image; in the test stage, the image is first preprocessed, specifically using global tone mapping [2], and then mean filtering. The calculation formula of the preprocessing method is:

[0013]

[0014] Where log represents the logarithmic function, max is the operation of taking the maximum value, Y represents the brightness channel of the input image, ε is a small value set to prevent division by 0, and the parameter λ is obtained by solving the optimization problem of formula (5):

[0015]

[0016] Among them, H l (Y c ) indicates Y c The histogram of l(LDR) is the average histogram of the brightness channel of a large number of high-quality natural LDR images.

[0017] In the training phase, the network structure described above is integrated into the Denoising Diffusion Probabilistic Models (DDPM) [5]; in the testing phase, the network structure described above is integrated into the Denoising Diffusion Implicit Models (DDIM) [6]. In these two diffusion models, the network designed by the present invention is used to predict the noise added at each step of the diffusion model.

[0018] (2) Based on the network model described in step (1), the structural information is input into the first-scale encoding module of the control module, and the brightness distribution map is input into the second-scale encoding module of the control module, so as to further enhance the detail information of the output image.

[0019] (3) Based on the DDIM mentioned in step (1), the intermediate results generated by each iteration or part of the iterations of its reverse sampling iterative process are corrected, that is, the hue information of the generated result is taken and combined with the structural information of the original image to further protect the structural information of the original image. The corrected result is then used for subsequent calculations in the reverse sampling process according to the formula of the diffusion model.

[0020] Specifically, for any iteration of DDIM reverse sampling, the initial estimate of the noise-free data z0 is first calculated using the formula of the original DDIM algorithm, which is:

[0021]

[0022] in, is the initial estimate of the latent space embedding z0 of the noise-free data obtained by DDIM in this iteration, z t is the result of the previous iteration, The weights predefined for DDIM, ∈ θ (z t ,t,c) is the network model described in step (1), t is the time step of the current iteration, and c is the conditional input of the network model.

[0023] Then, the structural information of the original image is used to Make corrections. The correction method is:

[0024]

[0025] in, Yes The operation of transferring from the latent space of the SD model to the image domain, TSD represents the tone-structure decomposition of the image, that is, the local Gaussian mean, local variance and structural information of the image are calculated according to formulas (1) to (3) in step (1). γ is a hyperparameter that controls the degree of image detail enhancement. is the structural information of the original image, and ε is the operation of transferring the image to the latent space of the SD model; After that, it is used as the final estimate of z0 in this iteration, and the subsequent calculations of this iteration are performed according to the principle of the original DDIM algorithm to obtain the result of this iteration. The calculation formula for the subsequent calculation of this iteration is:

[0026]

[0027] In the formula It is also the weight predefined by DDIM.

[0028] (4) Network training and testing. The network is trained on a high-quality LDR image dataset. During training, the parameters of the generative model are frozen and only the control module is trained. The loss function used is the same as that used in [4], [5], etc., namely, the L2 loss of the noise predicted by the network and the noise actually added to the noise-free image. After training, the model can be directly used for tone mapping of HDR images or other similar tasks.

[0029] During testing, only the brightness component of the image generated by the network is taken, and then converted into an RGB image using the following formula:

[0030]

[0031] Among them, Y pred and Y are the brightness components of the network generated image and the original image respectively, i∈{R,G,B}, and They represent one of the three RGB channels of the input image and the final output result respectively, and s is a hyperparameter.

[0032] The technical features and performance advantages of the present invention are mainly:

[0033] The network only needs to be trained with LDR images, and the obtained network model can be directly used for tone mapping of HDR images or other similar tasks without retraining on the image category to be processed, that is, the original image information can be well preserved and the HDR image tone mapping effect can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The figure is a flowchart of the high dynamic range image tone mapping method based on the diffusion model and zero-sample training of the present invention.

[0035] Figure 2 Illustration of the tone mapping network model structure designed for the present invention.

[0036] Figure 3 is a set of processing examples, in which from left to right are the HDR image after linear scaling of pixel values, the results output by references [3][1][2], and the results output by the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described below by way of embodiments in conjunction with the accompanying drawings.

[0038] The high dynamic range image tone mapping method based on diffusion model and zero-sample training provided by the present invention comprises the following specific steps:

[0039] Step 1: Build Figure 2 The network structure shown in Figure 1. TSD (Tone-Structure Decomposition) means decomposing an image into structural information and tone information, and TSR (Tone-Structure Reconstruction) means reconstructing structural information and tone information into an image. The pre-trained generative model uses the version v1.5 released by the SD model, which includes cascaded encoding modules of four scales, an intermediate module, and cascaded decoding modules of four scales. The control module includes encoding modules of four scales with the same structure as the SD model, but a zero convolution layer is added at the end of the neural network of each scale, and the output of each scale is added to the feature map of the corresponding scale output of the generative model.

[0040] The above network is integrated into the framework of the diffusion model. The reverse sampling algorithm of the diffusion model adopts DDIM[6], and the default reverse sampling step number is set to 20 steps (i.e., 20 iterations).

[0041] Step 2: The conditional input of the network includes structural information and a rough brightness distribution map. The structural information is calculated using equations (1) to (3), where ω uses a Gaussian filter kernel of 7×7 size and a standard deviation of 7 / 6, and ε is set to 0.0000001. The method for calculating the rough brightness distribution map is different in the training stage and the test stage. In the training stage, the calculation method is to perform a mean filter on the brightness channel of the image; in the test stage, it is necessary to first use equations (4) to (5) to obtain the result of the preliminary tone mapping of the brightness channel, and then perform a mean filter on the result. The hyperparameter λ in equation (4) defaults to 0.05. The size of the mean filter kernel can be appropriately set larger, such as 0.02 times the smaller value of the length and width of the original image. The structural information is input into the first layer of the first scale of the control module, and the rough brightness distribution map is input into the first layer of the second scale of the control module.

[0042] Step 3: Select some iterations in the DDIM reverse sampling process and use equations (6) to (9) to correct the results of the iterations. Under the default configuration, the results calculated for the first 10 iterations and the last iteration are corrected. The hyperparameter γ in equation (7) is set to 2 to enhance the texture of the image.

[0043] Step 4: Use the network structure described in step 1, use the input described in step 2, freeze the parameters of the SD model, and only train the parameters of the control module. The training set can use a high-quality LDR image dataset, such as DIV2K[7]. During testing, only the brightness component of the image generated by the network is taken, and then converted into an RGB image using formula (8), where the hyperparameter s is set to 0.5.

[0044] A set of examples of using the present invention to perform tone mapping on HDR images and comparison with other algorithm results Figure 3 Compared with other tone mapping methods, the image output by the present invention contains richer information in the overly bright and overly dark areas in the scene (framed in the figure and enlarged in the lower right corner of each figure), which has obvious advantages. Therefore, it is shown that the present invention is more suitable for HDR image tone mapping under extreme lighting conditions.

[0045] References

[0046] [1]. Aakanksha Rana, Praveer Singh, Giuseppe Valenzise, ​​Frederic Dufaux, Nikos Komodakis, and Aljosa Smolic. Deep tone mapping operator for high dynamic range images. IEEE Transactions on Image Processing, 29:1285–1298, 2020.

[0047] [2].Yael Vinker,Inbar Huberman-Spiegelglas,and Raanan Fattal.Unpairedlearning for high dynamic range image tone mapping.In 2021IEEE / CVFInternational Conference on Computer Vision(ICCV),pages 14637–14646,2021.

[0048] [3].Kai-Fu Yang,Cheng Cheng,Shi-Xuan Zhao,Hong-Mei Yan,Xian-ShiZhang,and Yong-Jie Li.Learning to adapt to light.Int.J.Comput.Vision,131(4):1022–1041,2023

[0049] [4].R.Rombach,A.Blattmann,D.Lorenz,P.Esser,and B.Ommer,“High-resolution image synthesis with latent diffusion models,”in Proceedings ofthe IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR),June2022,pp.10 684–10 695

[0050] [5].J.Ho,A.Jain,and P.Abbeel,“Denoising diffusion probabilisticmodels,”in Advances in Neural Information Processing Systems,H.Larochelle,M.Ranzato,R.Hadsell,M.Balcan,and H.Lin,Eds.,vol.33.Curran Associates,Inc.,2020,pp.6840–6851.[Online].Available:

[0051] https: / / proceedings.neurips.cc / paperfiles / paper / 2020 / file / 4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf

[0052] [6].J.Song,C.Meng,and S.Ermon,“Denoising diffusion implicit models,”arXiv preprint arXiv:2010.02502,2022

[0053] [7].E.Agustsson and R.Timofte,"NTIRE 2017Challenge on Single ImageSuper-Resolution:Dataset and Study,"2017IEEE Conference on Computer Visionand Pattern Recognition Workshops(CVPRW),Honolulu,HI,USA,2017,pp.1122-1131。

Claims

1. A high dynamic range image tone mapping method based on diffusion model and zero-shot training, It is characterized in that It includes building a tone mapping network model and integrating it into the framework of the diffusion model, taking the structural information and the approximate brightness distribution map as the conditional input of the diffusion model, and using the structural information of the original image to correct the intermediate result of the reverse sampling of the diffusion model; the specific steps are: (1) constructing a tone mapping network model for each step in the diffusion model; the network model structure includes a generation model for pre-training and a control module of the model; The generation model adopts the network structure of the stable diffusion model (SD), including: cascaded encoding modules of four scales, an intermediate module, and cascaded decoding modules of four scales; the structure of the control module includes the same encoding modules of four scales as the SD model, but a zero convolution layer is added at the end of the neural network of each scale; the output of each scale of the control module is added to the feature map of the corresponding scale output of the generation model, so as to control the final output result of the generation model; the input of the control module includes the structural information and the approximate brightness distribution map at each pixel of the input image; wherein: The calculation formula for the structural information at each pixel is: In the formula, I ori represents the original image, μ ori and σ ori They represent the local Gaussian mean and local standard deviation of the image respectively, ε is a small value introduced to prevent division by 0, (i, j) is the coordinate of the current pixel in the image; μ ori and σ ori The specific calculation formula is: Among them, K represents the window size of Gaussian filtering, and ω represents the Gaussian filtering kernel; The calculation method of the brightness distribution map is as follows: in the training stage, the brightness channel of the image is mean filtered; in the testing stage, the image is preprocessed, specifically using global tone mapping, and then mean filtered. The calculation formula for preprocessing is: Where log represents the logarithmic function, max is the operation of taking the maximum value, Y represents the brightness channel of the input image, ε is a small value set to prevent division by 0, and the parameter λ is obtained by solving the optimization problem of formula (5): Among them, H l (Y c ) indicates Y c The histogram of l (LDR) is the average histogram of the brightness channel of a large number of high-quality natural LDR images; In the training phase, the above network structure is integrated into the denoised diffusion probabilistic model (DDPM); in the testing phase, the above network structure is integrated into the denoised diffusion implicit model (DDIM); in both diffusion models, the network structure is used to predict the noise added at each step of the diffusion model; (2) Based on the network model described in step (1), the structural information is input into the first-scale encoding module of the control module, and the brightness distribution map is input into the second-scale encoding module of the control module to further enhance the detail information of the output image; (3) Based on the DDIM described in step (1), the intermediate results generated by each iteration or part of the iterations of its reverse sampling iterative process are corrected, that is, the hue information of the generated result is taken and combined with the structural information of the original image to further protect the structural information of the original image; the corrected result is then used for subsequent calculations in the reverse sampling process according to the formula of the diffusion model; Specifically, for any iteration of DDIM reverse sampling, the latent space feature embedding z of the noise-free data is first calculated using the formula of the original DDIM algorithm. 0 The initial estimate of is calculated as: in, is the latent space feature embedding z of the noise-free data obtained by DDIM in this iteration 0 The initial estimate of z t is the result of the previous iteration, The weights predefined for DDIM, ∈ θ (z t ,t,c) is the network model described in step (1), t is the time step of the current iteration, and c is the conditional input of the network model; Then, the structural information of the original image is used to Make corrections, the correction method is: in, Yes The operation of transferring from the latent space of the SD model to the image domain, TSD represents the tone-structure decomposition of the image, that is, the local Gaussian mean, local variance and structural information of the image are calculated according to formulas (1) to (3) in step (1). γ is a hyperparameter that controls the degree of image detail enhancement. is the structural information of the original image, and ε is the operation of transferring the image to the latent space of the SD model; Afterwards, it is used as the value of z in this iteration. 0 The final estimate is calculated according to the principle of the original DDIM algorithm to obtain the result of this iteration. The calculation formula for the subsequent calculation of this iteration is: In the formula It is also the weight predefined by DDIM; (4) Network training and testing; The network is trained on the LDR image dataset; During training, the parameters of the generative model are frozen and only the control module is trained; The loss function is the L2 loss of the noise predicted by the network and the noise actually added to the noise-free image; The trained network model is directly used for tone mapping of HDR images or other similar tasks; During testing, only the brightness component of the image generated by the network is taken, and then converted into an RGB image using the following formula: Among them, Y pred and Y are the brightness components of the network generated image and the original image respectively, i∈{R,G,B}, and They represent one of the three RGB channels of the input image and the final output result respectively, and s is a hyperparameter.

Citation Information

Patent Citations

  • Image high dynamic range reconstruction method based on deep learning

    CN111292264A

  • Image processing method and related device

    CN117974493A

  • Methods and Apparatus for Tone Mapping High Dynamic Range Images

    US20130121572A1