Diffusion model-based multi-exposure-image high-dynamic-range image reconstruction method

Abstract

The invention discloses a multi-exposure image high dynamic range image reconstruction method based on a diffusion model, belongs to the technical field of computer vision, and adopts forward noise addition and reverse noise reduction diffusion processing on a high dynamic range image. In order to restore details of an overexposure area and an underexposure area and avoid introduction of motion artifacts, a multi-exposure high-dynamic image restoration framework of a deep neural network based on a diffusion model is adopted, and information of a low-dynamic-range image is fully utilized in combination with a space attention mechanism and an interactive attention mechanism. And the consistency and high fidelity of the image content are ensured. The method is used for reconstructing the high dynamic range image.

Description

Technical Field

[0001] The present invention belongs to the field of underlying visual technology, and specifically relates to a method for reconstructing high dynamic range images of multiple exposure images based on a diffusion model. Background Art

[0002] Natural brightness values ​​have a wide visual dynamic range. However, digital photography sensors usually capture images with limited illumination variations, resulting in low dynamic range photos. Therefore, low dynamic range images usually contain overexposed or underexposed areas and cannot meet human visual expectations of brightness and darkness. Multi-exposure fusion is able to reconstruct high dynamic range images using information from low dynamic range images at different exposures.

[0003] When there is object motion or camera motion, the necessary information of the overexposed position in the reference frame may not be available in other low dynamic range images. Since deep neural network-based methods rely on applying L1 or L2 losses to promote the network to fit the complex relationship between low dynamic range images and true high dynamic range images, satisfactory high dynamic range images cannot be produced when motion and saturation are present at the same time. Methods based on generative adversarial networks can generate more realistic details for areas where information is missing. Combined with adversarial loss, high dynamic range images can be generated to produce missing content when low dynamic range images have large overexposed areas. Although adversarial loss can alleviate this problem, these methods may overfit certain visual features or data distributions due to training problems, and may generate unreasonable content and artifacts. Based on the above, the present invention proposes a high dynamic range image reconstruction method for multiple exposure images based on a diffusion model. Summary of the invention

[0004] The purpose of the present invention is to propose a multi-exposure image high dynamic range image reconstruction method based on a diffusion model to solve the problem that it is difficult to generate reliable content when a low dynamic range image contains a large range of over-exposed or under-exposed areas.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A method for reconstructing a high dynamic range image of multiple exposure images based on a diffusion model comprises the following steps:

[0007] S1, the diffusion process of constructing a diffusion model for high dynamic range image reconstruction;

[0008] S2, establishing a high dynamic range image reconstruction diffusion model, taking the multi-exposure low dynamic range image as input, and reconstructing the output of the high dynamic range image diffusion model to predict the noise;

[0009] S3, training a high dynamic range image reconstruction diffusion model based on the diffusion process constructed in S1;

[0010] S4. Based on the inverse process of the diffusion model, the high dynamic range image is iteratively reconstructed using the predicted noise.

[0011] Preferably, the S1 specifically includes the following contents:

[0012] Gaussian noise is repeatedly added to the high dynamic range image through a diffusion process, assuming that the noise addition process conforms to the Gaussian distribution:

[0013]

[0014] Among them, y t and t-1 denote the high dynamic range noise images at step t and step t-1 respectively; q(y t |y t-1 ) is y t Towards y t-1 The diffusion process, α t Used to determine the variance of the noise added in each iteration, the value range is 0<α t <1;

[0015] According to the normal distribution superposition, the high dynamic range image y0 is diffused to y t The intermediate step of edge processing is performed to obtain the t-th step high dynamic range noise image y under the given high dynamic range image y0. t The conditional probability distribution of :

[0016]

[0017] in, According to this distribution, a high dynamic range noise image y is obtained by resampling the parameters t The expression is:

[0018]

[0019] Among them, ∈ represents a random noise image sampled from a normal distribution with a mean of 0 and a variance of 1;

[0020] The whole diffusion process takes the high dynamic range image y0 as input, and after a continuous t-step diffusion process, outputs a standard Gaussian noise image y T ; Where t = 1, 2, …, T, T represents the total number of input high and low resolution image data sets and diffusion steps, T = 1000.

[0021] Preferably, the high dynamic range image reconstruction diffusion model includes the following two parts:

[0022] ①Multi-exposure image feature guidance network g based on spatial attention mechanismθ :The multi-exposure image feature guide network g θ The method is composed of a spatial attention-based module, which takes the reference frame I2 and the adjacent frames I1 and I3 as inputs, firstly extracts image features through convolution layers, concatenates the reference image features and the adjacent frame features along the feature channel direction, and inputs multiple cascaded convolution-activation layers to generate a set of spatial weights, and multiplies the spatial weights with the adjacent frames to obtain the alignment features of the adjacent frames; then the alignment features are transformed through linear layers, and spatial dynamic affine transformation parameters are generated through multiple convolution layers; the spatial dynamic affine transformation parameters include scaling parameters and offset parameters; finally, the features of the reference frame are modulated using the affine transformation parameters to obtain modulation enhancement features;

[0023] ②Conditional noise prediction network ε based on interactive attention mechanism θ ; It consists of a U-Net network based on pixel-level cross attention, which first downsamples the input features and then restores the original feature size through upsampling; in each layer of the U-Net network, it includes a structure of convolution-window-based pixel-level interactive attention-convolution, and residual connections are added to each layer of the network; among them, the specific contents of the window-based pixel-level interactive attention include the following:

[0024] The reference frame I2 is passed through a convolutional layer to extract low-dynamic reference features; then, the low-dynamic reference features and modulation enhancement features are divided into multiple local windows of fixed size; the pixel features in the low-dynamic reference feature window are used as the query vector, and the pixel features in the corresponding window of the modulation enhancement feature are used as the key and value vectors; then the dot product between the query vector and the key vector is calculated, and the result is normalized to obtain the attention score; finally, the value vector is weighted and summed using the attention score to obtain the enhanced representation of the reference feature.

[0025] Preferably, S3 specifically includes the following contents:

[0026] The neural network parameters of the high dynamic range image reconstruction diffusion model are randomly initialized, and a gradient descent operation is performed according to the loss function, wherein the loss function specifically refers to:

[0027] ||ε-ε θ (y t ,g θ (I1.I2,I3),t)||ε~N(0,1)

[0028] Where ||·|| represents the L1 paradigm; t represents any random number in {1, 2, …, T}; N(0,1) represents a normal distribution with a mean of 0 and a variance of 1; ε~N(0,1) means that ε obeys N(0,1). Through gradient backpropagation, the multi-exposure image feature guidance network g based on the spatial attention mechanism is jointly trained. θand the conditional noise prediction network ε based on the interactive attention mechanism θ ;

[0029] The parameters of the high dynamic range image reconstruction diffusion model are optimized, and the neural network is iteratively updated until the loss function converges and the high dynamic range image reconstruction diffusion model training is completed.

[0030] Preferably, the S4 specifically includes the following contents:

[0031] The inference process of the high dynamic range image reconstruction diffusion model is defined as a reverse Markov process, which is the inverse process of the forward diffusion process. T Start, y T Obey Gaussian distribution:

[0032] y T ~N(0,1)

[0033] The predicted high dynamic range image with lower noise is obtained by step-by-step reasoning, where the step-by-step reasoning process conforms to the state transition of Markov and obeys the conditional Gaussian distribution:

[0034]

[0035] Among them, p θ (yt-1|yt,x) represents a learnable conditional Gaussian distribution; μ θ (I1,I2,I3,y t ,γ t ) represents the mean of the learnable distribution; Represents the distribution variance, which is set to the default value 1-α t ;

[0036] Assume that given y t and y0,y t-1 The posterior distribution of is a conditional Gaussian distribution:

[0037]

[0038] Derived from the Bayesian formula t-1 The mean of the posterior distribution q is:

[0039]

[0040] The distribution p learned by the neural network θ Fit the posterior distribution, estimate the noise ε through the high dynamic range image reconstruction diffusion model, and obtain an approximate high dynamic range image through transformation

[0041]

[0042] Then we can get the learnable distribution p θ The mean of is:

[0043]

[0044] The diffusion model for high dynamic range image reconstruction is finally inferred step by step through re-parameter sampling:

[0045]

[0046] Among them, ε~N(0,1), the whole inverse process is carried out for T steps, T=1000, that is, t=T,T-1,…,1; the initial inverse step inputs the Gaussian noise image, the low dynamic range image, and the time t of the current inverse step into the high dynamic range image reconstruction diffusion model, and the inference result is input into the diffusion model of the next step. After T inverse steps, the final high dynamic range reconstructed image is obtained.

[0047] Compared with the prior art, the present invention provides a multi-exposure image high dynamic range image reconstruction method based on a diffusion model, which has the following beneficial effects:

[0048] The high dynamic range image reconstruction method proposed in the present invention can generate reasonable detail information in over-exposed areas and under-exposed areas, and can effectively avoid the introduction of motion artifacts. Through the multi-exposure image feature guidance network based on the spatial attention mechanism and the conditional noise prediction network based on the interactive attention mechanism, the guidance information of the low dynamic range image can be efficiently utilized, ensuring the consistency and high fidelity of the generated content. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the effects of forward denoising diffusion and reverse denoising diffusion of high dynamic range images of the present invention.

[0050] Figure 2 Schematic diagram of the high dynamic range image reconstruction process of multiple exposure images based on the diffusion model of the present invention.

[0051] Figure 3 Schematic diagram of the multi-exposure image feature guidance network based on the spatial attention mechanism and the window-based pixel-level interactive attention mechanism in the present invention.

[0052] Figure 4 The results of the present invention are compared with other high dynamic range image reconstruction methods. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0054] Embodiment 1:

[0055] See also Figure 1 The present invention proposes a multi-exposure image high dynamic range image reconstruction method based on a diffusion model, comprising the following steps:

[0056] S1, establishing the diffusion process of the diffusion model for high dynamic range image reconstruction;

[0057] Gaussian noise is repeatedly added to the high dynamic range image through a diffusion process, assuming that the noise addition process conforms to the Gaussian distribution:

[0058]

[0059] Among them, y t and t-1 are the high dynamic range noise images at step t and step t-1 respectively, q(y t |y t-1 ) is y t Towards y t-1 The diffusion process, α t Used to determine the variance of the noise added in each iteration, the value range is 0<α t <1.

[0060] According to the superposition of normal distribution, the high dynamic range image y0 can be diffused to y t The intermediate step of edge processing is performed to obtain the t-th step high dynamic range noise image y under the given high dynamic range image y0. t The conditional probability distribution of :

[0061]

[0062] in, According to this distribution, a high dynamic range noise image y can be obtained by resampling the parameters t The expression is:

[0063]

[0064] Here, ∈ represents a random noise image sampled from a normal distribution with a mean of 0 and a variance of 1.

[0065] The whole diffusion process takes the high dynamic range image y0 as input, and after a continuous t-step diffusion process, t = 1, 2, ..., T, T = 1000, outputs a standard Gaussian noise image y T .

[0066] S2, establishing a high dynamic range image reconstruction diffusion model, taking the multi-exposure low dynamic range image as input, and the high dynamic range image reconstruction diffusion model outputs predicted noise;

[0067] The diffusion model for high dynamic range image reconstruction is mainly divided into the following two parts:

[0068] ①Multi-exposure image feature guidance network g based on spatial attention mechanism θ :It is mainly composed of a spatial attention module: the reference frame I2 and adjacent frames I1 and I3 are used as inputs. First, the image features are extracted separately through the convolution layer, and the reference image features and adjacent frame features are spliced ​​along the feature channel direction. The reference image features and adjacent frame features are input into multiple cascaded convolution-activation layers to generate a set of spatial weights, and the spatial weights are multiplied with the adjacent frames to obtain the alignment features of the adjacent frames; then the alignment features are transformed through the linear layer, and then through multiple convolution layers to generate spatial dynamic affine transformation parameters, including scaling parameters and offset parameters. The affine transformation parameters are then used to modulate the features of the reference frame to obtain modulation enhancement features.

[0069] ②Conditional noise prediction network ε based on interactive attention mechanism θ :It is mainly composed of a U-Net type network based on pixel-level cross attention: first, the input features are downsampled, and then the original feature size is restored by upsampling. In each layer of the U-Net network, it mainly includes the structure of convolution-window-based pixel-level interactive attention-convolution, and residual connections are added to each layer of the network. Among them, the specific steps of window-based pixel-level interactive attention are as follows: The reference frame I2 is extracted through the convolution layer. Subsequently, the low-dynamic reference features and modulation enhancement features are divided into multiple local windows of fixed size. The pixel features in the low-dynamic reference feature window are used as the query vector, and the pixel features in the corresponding window of the modulation enhancement feature are used as the key and value vectors. The attention score is obtained by calculating the dot product between the query vector and the key vector and normalizing the result. Finally, the value vector is weighted summed using the attention score to obtain the enhanced representation of the reference feature.

[0070] S3, training a diffusion model for high dynamic range image reconstruction based on a diffusion process;

[0071] The high dynamic range image reconstruction diffusion model is trained, and the training steps are:

[0072] Input high and low resolution image data sets and the total number of diffusion steps T, where y0 is a high dynamic range image, I2 is its corresponding low dynamic range image, and I1 and I3 are adjacent frames of I2 respectively.

[0073] The neural network parameters of the high dynamic range image reconstruction diffusion model are randomly initialized, and the gradient descent operation is performed according to the loss function. The loss function is as follows:

[0074] ||ε-ε θ (y t ,gθ (I1.I2,I3),t)||ε~N(0,1)

[0075] Where ||·|| represents the L1 paradigm, t randomly selects a number from {1, 2, …, T}, and N(0,1) is a normal distribution with a mean of 0 and a variance of 1. Through gradient back propagation, the multi-exposure image feature guidance network g based on the spatial attention mechanism can be jointly trained. θ and the conditional noise prediction network ε based on the interactive attention mechanism θ .

[0076] The parameters of the high dynamic range image reconstruction diffusion model are optimized, and the neural network is iteratively updated until the loss function converges and the high dynamic range image reconstruction diffusion model training is completed.

[0077] S4, based on the inverse process of the diffusion model, using the predicted noise, iteratively reconstructs the high dynamic range image;

[0078] The inference process of the high dynamic range image reconstruction diffusion model can be defined as a reverse Markov process, which is the inverse process of the forward diffusion process. T Start, y T Obey Gaussian distribution:

[0079] y T ~N(0,1)

[0080] The predicted high dynamic range image with lower noise is obtained by step-by-step reasoning, where the step-by-step reasoning process conforms to the state transition of Markov and obeys the conditional Gaussian distribution:

[0081]

[0082] Among them, p θ (y t-1 |y t ,x) indicates that it contains a learnable conditional Gaussian distribution, which is specifically reflected in μ θ (I1,I2,I3,y t ,γ t ) is the mean of the learnable distribution, is the distribution variance, set to the default value 1-α t .

[0083] Assume that given y t and y0,y t-1 The posterior distribution of is a conditional Gaussian distribution:

[0084]

[0085] By using the Bayesian formula, we can deduce y t-1The mean of the posterior distribution q is:

[0086]

[0087] The distribution p that can be learned by a neural network θ Fit the posterior distribution, specifically, estimate the noise ε through the high dynamic range image reconstruction diffusion model. Obtain an approximate high dynamic range image by transforming

[0088]

[0089] So we can get the learnable distribution p θ The mean of is:

[0090]

[0091] The diffusion model for high dynamic range image reconstruction can finally be inferred step by step by re-parameter sampling:

[0092]

[0093] The entire inverse process is performed for T steps, T = 1000, that is, t = T, T-1, ..., 1. The initial inverse step inputs the Gaussian noise image, the low dynamic range image, and the time t of the current inverse step into the high dynamic range image reconstruction diffusion model, and the inference result is input into the diffusion model of the next step. After T inverse steps, the final high dynamic range reconstructed image is obtained.

[0094] In summary, the present invention proposes a multi-exposure image high dynamic range image reconstruction method based on a diffusion model, which adopts forward denoising and reverse denoising diffusion processing for high dynamic range images, and simultaneously utilizes effective network guidance strategy and interaction strategy to realize multi-exposure image reconstruction of high dynamic range images.

[0095] Embodiment 2:

[0096] See also Figure 4 , based on Example 1 but different in that,

[0097] The advanced comparison methods selected by this invention include: AHDR, HDRGAN, where AHDR is a method based on L1 loss constraint, and HDRGAN is a method that uses GAN to reconstruct high dynamic range images. For these methods, we tested them on the same dataset and obtained Figure 4 The effect diagram shown.

[0098] from Figure 4 It can be seen that the multi-exposure image high dynamic range image reconstruction method based on the diffusion model proposed in the present invention can effectively alleviate the motion artifact problem and generate semantically consistent details in the over-exposed area.

[0099] Table 1 is a table of comparative experimental results of various comparison methods. μ and L represent the calculation indicators in the tone mapping domain and the linear domain, where PSNR is the peak signal-to-noise ratio indicator, SSIM measures the structural similarity, and FID is the perceptual indicator.

[0100] Table 1

[0101]

[0102]

[0103] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for reconstructing high dynamic range images from multi-exposure images based on a diffusion model, characterized in that: The following steps are involved: S1, the diffusion process of constructing a diffusion model for high dynamic range image reconstruction; S2, establishing a high dynamic range image reconstruction diffusion model, taking the multi-exposure low dynamic range image as input, and reconstructing the output of the high dynamic range image diffusion model to predict the noise; S3, training a high dynamic range image reconstruction diffusion model based on the diffusion process constructed in S1; S4. Based on the inverse process of the diffusion model, the high dynamic range image is iteratively reconstructed using the predicted noise.

2. The method for reconstructing a high dynamic range image from a multi-exposure image based on a diffusion model according to claim 1, characterized in that: The S1 specifically includes the following contents: Gaussian noise is repeatedly added to the high dynamic range image through a diffusion process, assuming that the noise addition process conforms to the Gaussian distribution: Among them, y t and t-1 denote the high dynamic range noise images at step t and step t-1 respectively; q(y t |y t-1 ) is y t Towards y t-1 The diffusion process, α t Used to determine the variance of the noise added in each iteration, the value range is 0<α t <1; According to the normal distribution superposition, the high dynamic range image y0 is diffused to y t The intermediate step of edge processing is performed to obtain the t-th step high dynamic range noise image y under the given high dynamic range image y0. t The conditional probability distribution of : in, According to this distribution, a high dynamic range noise image y is obtained by resampling the parameters t The expression is: Where, ∈ represents a random noise image sampled from a normal distribution with a mean of 0 and a variance of 1; The whole diffusion process takes the high dynamic range image y0 as input, and after a continuous t-step diffusion process, outputs a standard Gaussian noise image y T ; Where t = 1, 2, …, T, T represents the total number of input high and low resolution image data sets and diffusion steps, T = 1000.

3. The method for reconstructing a multi-exposure image with a high dynamic range based on a diffusion model according to claim 2, characterized in that: The high dynamic range image reconstruction diffusion model includes the following two parts: ①Multi-exposure image feature guidance network g based on spatial attention mechanism θ :The multi-exposure image feature guide network g θ It is composed of a spatial attention-based module. It takes the reference frame I2 and adjacent frames I1 and I3 as inputs. First, the image features are extracted through the convolution layer. The reference image features and adjacent frame features are concatenated along the feature channel direction. The reference image features and adjacent frame features are input into multiple cascaded convolution-activation layers to generate a set of spatial weights. The spatial weights are multiplied with the adjacent frames to obtain the alignment features of the adjacent frames. Then the alignment features are transformed through a linear layer, and spatial dynamic affine transformation parameters are generated through multiple convolutional layers; the spatial dynamic affine transformation parameters include scaling parameters and offset parameters; finally, the features of the reference frame are modulated using the affine transformation parameters to obtain modulation enhancement features; ②Conditional noise prediction network ε based on interactive attention mechanism θ :It is composed of a U-Net type network based on pixel-level cross attention. The input features are first downsampled, and then the original feature size is restored by upsampling. In each layer of the U-Net network, it includes a structure of convolution-window-based pixel-level interactive attention-convolution, and residual connections are added to each layer of the network. Among them, the specific contents of the window-based pixel-level interactive attention include the following: The reference frame I2 is passed through a convolutional layer to extract low-dynamic reference features; then, the low-dynamic reference features and modulation enhancement features are divided into multiple local windows of fixed size; the pixel features in the low-dynamic reference feature window are used as the query vector, and the pixel features in the corresponding window of the modulation enhancement feature are used as the key and value vectors; then the dot product between the query vector and the key vector is calculated, and the result is normalized to obtain the attention score; finally, the value vector is weighted and summed using the attention score to obtain the enhanced representation of the reference feature.

4. The method for reconstructing a high dynamic range image from multiple exposure images based on a diffusion model according to claim 3, characterized in that: The S3 specifically includes the following contents: The neural network parameters of the high dynamic range image reconstruction diffusion model are randomly initialized, and a gradient descent operation is performed according to the loss function, wherein the loss function specifically refers to: ||ε-ε θ (y t ,g θ (I1.I2,I3),t)||ε~N(0,1) Where ||·|| represents the L1 paradigm; t represents any random number in {1, 2, …, T}; N(0,1) represents a normal distribution with a mean of 0 and a variance of 1; ε~N(0,1) means that ε obeys N(0,1). Through gradient backpropagation, the multi-exposure image feature guidance network g based on the spatial attention mechanism is jointly trained. θ and the conditional noise prediction network ε based on the interactive attention mechanism θ ; The parameters of the high dynamic range image reconstruction diffusion model are optimized, and the neural network is iteratively updated until the loss function converges and the high dynamic range image reconstruction diffusion model training is completed.

5. The method for reconstructing a high dynamic range image from multiple exposure images based on a diffusion model according to claim 4, characterized in that: The S4 specifically includes the following contents: The inference process of the high dynamic range image reconstruction diffusion model is defined as a reverse Markov process, which is the inverse process of the forward diffusion process. T Start, y T Obey Gaussian distribution: and T ~N(0,1) The predicted high dynamic range image with lower noise is obtained by step-by-step reasoning, where the step-by-step reasoning process conforms to the state transition of Markov and obeys the conditional Gaussian distribution: Among them, p θ (yt-1|yt,x) represents a learnable conditional Gaussian distribution; μ θ (I1,I2,I3,y t ,γ t ) represents the mean of the learnable distribution; Represents the distribution variance, which is set to the default value 1-α t ; Assume that given y t and y0,y t-1 The posterior distribution of is a conditional Gaussian distribution: Derived from the Bayesian formula t-1 The mean of the posterior distribution q is: The distribution p learned by the neural network θ Fit the posterior distribution, estimate the noise ε through the high dynamic range image reconstruction diffusion model, and obtain an approximate high dynamic range image through transformation Then we can get the learnable distribution p θ The mean of is: The diffusion model for high dynamic range image reconstruction is finally inferred step by step through re-parameter sampling: Among them, ε~N(0,1), the whole inverse process is carried out for T steps, T=1000, that is, t=T,T-1,…,1; the initial inverse step inputs the Gaussian noise image, the low dynamic range image, and the time t of the current inverse step into the high dynamic range image reconstruction diffusion model, and the inference result is input into the diffusion model of the next step. After T inverse steps, the final high dynamic range reconstructed image is obtained.

Images