A multi-exposure image high dynamic range image reconstruction method based on a diffusion model

Abstract

The application discloses a multi-exposure image high dynamic range image reconstruction method based on a diffusion model, belongs to the technical field of computer vision, and adopts forward noise adding and reverse noise removing diffusion processing on a high dynamic range image. In order to restore details of overexposed areas and underexposed areas and simultaneously avoid introducing motion artifacts, the application adopts a multi-exposure high dynamic image restoration framework based on a deep neural network of the diffusion model, simultaneously combines a spatial attention mechanism and an interactive attention mechanism to fully utilize information of a low dynamic range image, and guarantees consistency and high fidelity of image content. The application is used for high dynamic range image reconstruction.

Description

Technical Field

[0001] The present invention belongs to the field of underlying visual technology, and specifically relates to a method for reconstructing high dynamic range images from multi-exposure images based on a diffusion model. Background Art

[0002] Natural brightness values ​​have a wide visual dynamic range. However, digital photography sensors typically capture images with limited illumination variations, resulting in low dynamic range images. As a result, low dynamic range images often contain overexposed or underexposed areas, failing to meet human visual expectations of brightness and darkness. Multi-exposure fusion can reconstruct a high dynamic range image using information from low dynamic range images taken at different exposures.

[0003] When there is object motion or camera motion, the necessary information of the overexposed position in the reference frame may not be available in other low dynamic range images. Since deep neural network based methods rely on applying L1 or L2 losses to promote the network to fit the complex relationship between low dynamic range images and true high dynamic range images, satisfactory high dynamic range images cannot be produced when motion and saturation are present at the same time. Methods based on generative adversarial networks can generate more realistic details for areas where information is missing. Combined with adversarial loss, high dynamic range images can be generated to produce missing content when low dynamic range images have large overexposed areas. Although adversarial loss can alleviate this problem, these methods may overfit certain visual features or data distributions due to training problems, and may generate unreasonable content and artifacts. Based on the above, the present invention proposes a high dynamic range image reconstruction method for multiple exposure images based on a diffusion model. Summary of the Invention

[0004] The purpose of the present invention is to propose a multi-exposure image high dynamic range image reconstruction method based on a diffusion model to solve the problem that it is difficult to generate reliable content when a low dynamic range image contains a large range of overexposed or underexposed areas.

[0005] The object of the present invention is achieved through the following technical solutions:

[0006] A method for reconstructing a high dynamic range image from a multi-exposure image based on a diffusion model comprises the following steps:

[0007] S1, construction of diffusion process for high dynamic range image reconstruction diffusion model;

[0008] S2. Establish a high dynamic range image reconstruction diffusion model, take the multi-exposure low dynamic range image as input, and output the predicted noise through the high dynamic range image reconstruction diffusion model;

[0009] S3, training the high dynamic range image reconstruction diffusion model based on the diffusion process constructed in S1;

[0010] S4, based on the inverse process of the diffusion model, iteratively reconstruct the high dynamic range image with the predicted noise.

[0011] Preferably, the S1 specifically comprises the following contents:

[0012] Repeatedly add Gaussian noise to the high dynamic range image through the diffusion process, assuming that the noise adding process conforms to the Gaussian distribution:

[0013]

[0014] Wherein, y t and y t-1 respectively represent the t-th step and the t-1-th step high dynamic range noise image; q(y t |y t-1 ) is the diffusion process of y t to y t-1 , α t is used to determine the variance of the noise added in each iteration, and the value range is 0 < α t < 1;

[0015] According to the normal distribution superposition, the high dynamic range image y0 is diffused to the intermediate step of y t , and the edge processing is carried out to obtain the conditional probability distribution of the t-th step high dynamic range noise image y t under the condition of the given high dynamic range image y0:

[0016]

[0017] Wherein, According to the distribution, the expression of the high dynamic range noise image y t is obtained by reparameterization sampling:

[0018]

[0019] Wherein, ∈ represents the random noise image sampled from the normal distribution with mean 0 and variance 1;

[0020] The whole diffusion process takes the high dynamic range image y0 as the input, and after the t-step diffusion process carried out in sequence, the noise image y T in the standard Gaussian form is output; wherein t = 1, 2, …, T, T represents the total number of input high and low resolution image data set and diffusion step, T = 1000.

[0021] Preferably, the high dynamic range image reconstruction diffusion model comprises the following two parts:

[0022] ① Multi-exposure image feature guided network g based on spatial attention mechanismθ :The multi-exposure image feature guide network g θ The method is composed of a spatial attention-based module. It takes the reference frame I2 and adjacent frames I1 and I3 as inputs. First, the image features are extracted separately through the convolution layer. The reference image features and the adjacent frame features are spliced ​​along the feature channel direction. The reference image features and the adjacent frame features are input into multiple cascaded convolution-activation layers to generate a set of spatial weights. The spatial weights are multiplied with the adjacent frames to obtain the alignment features of the adjacent frames. The aligned features are then transformed through the linear layer and passed through multiple convolution layers to generate spatial dynamic affine transformation parameters. The spatial dynamic affine transformation parameters include scaling parameters and offset parameters. Finally, the affine transformation parameters are used to modulate the features of the reference frame to obtain modulation enhancement features.

[0023] ②Conditional noise prediction network ε based on interactive attention mechanism θ It consists of a U-Net-type network based on pixel-level cross attention, which first downsamples the input features and then restores the original feature size through upsampling. In each layer of the U-Net network, it includes a convolution-window-based pixel-level interactive attention-convolution structure, and adds residual connections in each layer. The specific content of the window-based pixel-level interactive attention includes the following:

[0024] The reference frame I2 is passed through a convolutional layer to extract low-dynamic reference features. Subsequently, the low-dynamic reference features and modulation enhancement features are divided into multiple local windows of fixed size. The pixel features within the low-dynamic reference feature window are used as the query vector, and the pixel features within the corresponding window of the modulation enhancement feature are used as the key and value vectors. The dot product between the query vector and the key vector is then calculated, and the result is normalized to obtain the attention score. Finally, the attention score is used to perform weighted summation on the value vector to obtain an enhanced representation of the reference feature.

[0025] Preferably, the S3 specifically includes the following contents:

[0026] The neural network parameters of the diffusion model for high dynamic range image reconstruction are randomly initialized, and a gradient descent operation is performed according to the loss function, wherein the loss function is specifically:

[0027] ||ε-ε θ (y t ,g θ (I1.I2,I3),t)||ε~N(0,1)

[0028] Where ||·|| represents the L1 norm; t represents any random number in {1, 2, …, T}; N(0,1) represents a normal distribution with mean 0 and variance 1; ε~N(0,1) means that ε obeys N(0,1). Through gradient backpropagation, the multi-exposure image feature guidance network g based on the spatial attention mechanism is jointly trained. θand the conditional noise prediction network ε based on the interactive attention mechanism θ ;

[0029] The parameters of the high dynamic range image reconstruction diffusion model are optimized, and the neural network is iteratively updated until the loss function converges, and the training of the high dynamic range image reconstruction diffusion model is completed.

[0030] Preferably, the S4 specifically includes the following contents:

[0031] The inference process of the diffusion model for high dynamic range image reconstruction is defined as a reverse Markov process, which is the inverse process of the forward diffusion process. T Start, y T Obeying Gaussian distribution:

[0032] y T ~N(0,1)

[0033] A predicted high dynamic range image with lower noise is obtained through step-by-step reasoning, where the step-by-step reasoning process conforms to the state transition of Markov and obeys the conditional Gaussian distribution:

[0034]

[0035] Among them, p θ (yt-1|yt,x) represents a learnable conditional Gaussian distribution; μ θ (I1,I2,I3,y t ,γ t ) represents the mean of the learnable distribution; Represents the distribution variance, which is set to the default value 1-α t ;

[0036] Assume that given y t and y0,y t-1 The posterior distribution of is a conditional Gaussian distribution:

[0037]

[0038] Derived by Bayesian formula t-1 The mean of the posterior distribution q:

[0039]

[0040] The distribution p learned by the neural network θ Fit the posterior distribution, estimate the noise ε through the high dynamic range image reconstruction diffusion model, and obtain an approximate high dynamic range image through transformation

[0041]

[0042] Then we get the learnable distribution p θ The mean of is:

[0043]

[0044] The diffusion model for high dynamic range image reconstruction finally performs step-by-step inference via reparameter sampling:

[0045]

[0046] Among them, ε~N(0,1), the entire inverse process is carried out for T steps, T=1000, that is, t=T,T-1,…,1; the initial inverse step inputs the Gaussian noise image, the low dynamic range image, and the time t of the current inverse step into the high dynamic range image reconstruction diffusion model, and the inference result is input into the diffusion model of the next step. After T inverse steps, the final high dynamic range reconstructed image is obtained.

[0047] Compared with the prior art, the present invention provides a method for reconstructing high dynamic range images from multi-exposure images based on a diffusion model, which has the following beneficial effects:

[0048] The high dynamic range image reconstruction method proposed in this invention can generate reasonable detail information in overexposed areas and poorly exposed areas, while effectively avoiding the introduction of motion artifacts. Through the multi-exposure image feature guidance network based on the spatial attention mechanism and the conditional noise prediction network based on the interactive attention mechanism, the guidance information of the low dynamic range image can be efficiently utilized, ensuring the consistency and high fidelity of the generated content. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the effects of forward denoising diffusion and reverse denoising diffusion for high dynamic range images according to the present invention.

[0050] Figure 2 Schematic diagram of the high dynamic range image reconstruction process of multi-exposure images based on the diffusion model of the present invention.

[0051] Figure 3 Schematic diagram of the multi-exposure image feature guidance network based on the spatial attention mechanism and the window-based pixel-level interactive attention mechanism in the present invention.

[0052] Figure 4 The results of the present invention are compared with other high dynamic range image reconstruction methods. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0054] Example 1:

[0055] See also Figure 1 The present invention proposes a multi-exposure image high dynamic range image reconstruction method based on a diffusion model, comprising the following steps:

[0056] S1, establishing the diffusion process of the diffusion model for high dynamic range image reconstruction;

[0057] Gaussian noise is repeatedly added to the high dynamic range image through a diffusion process, assuming that the noise addition process conforms to the Gaussian distribution:

[0058]

[0059] Among them, y t and y t-1 are the high dynamic range noise images at step t and step t-1 respectively, q(y t |y t-1 ) is y t Towards y t-1 The diffusion process, α t Used to determine the variance of the noise added in each iteration, the value range is 0<α t <1.

[0060] According to the superposition of normal distribution, the high dynamic range image y0 can be diffused to y t The intermediate step of edge processing is performed to obtain the t-th step high dynamic range noise image y under the given high dynamic range image y0 t The conditional probability distribution of :

[0061]

[0062] in, According to this distribution, a high dynamic range noise image y can be obtained by re-sampling the parameters t The expression:

[0063]

[0064] Here, ∈ represents a random noise image sampled from a normal distribution with mean 0 and variance 1.

[0065] The whole diffusion process takes the high dynamic range image y0 as input, and after a series of t-step diffusion processes, t = 1, 2, ..., T, T = 1000, outputs a standard Gaussian noise image y T .

[0066] S2. Establish a high dynamic range image reconstruction diffusion model, take the multi-exposure low dynamic range image as input, and output the predicted noise of the high dynamic range image reconstruction diffusion model;

[0067] The diffusion model for high dynamic range image reconstruction is mainly divided into the following two parts:

[0068] ①Multi-exposure image feature guidance network g based on spatial attention mechanism θ The proposed method mainly consists of a spatial attention module: the reference frame I2 and adjacent frames I1 and I3 are taken as input. The image features are first extracted separately through the convolutional layer. The reference image features and adjacent frame features are spliced ​​along the feature channel direction. The reference image features and adjacent frame features are input into multiple cascaded convolution-activation layers to generate a set of spatial weights. The spatial weights are multiplied with the adjacent frames to obtain the aligned features of the adjacent frames. The aligned features are then transformed through the linear layer and then passed through multiple convolutional layers to generate spatial dynamic affine transformation parameters, including scaling parameters and offset parameters. The affine transformation parameters are then used to modulate the features of the reference frame to obtain modulation enhancement features.

[0069] ②Conditional noise prediction network ε based on interactive attention mechanism θ :It is mainly composed of a U-Net type network based on pixel-level cross attention: the input features are first downsampled, and then the original feature size is restored by upsampling. In each layer of the U-Net network, it mainly includes the structure of convolution-window-based pixel-level interactive attention-convolution, and residual connections are added to each layer of the network. Among them, the specific steps of window-based pixel-level interactive attention are as follows: the reference frame I2 is subjected to the convolution layer to extract low-dynamic reference features. Subsequently, the low-dynamic reference features and modulation enhancement features are divided into multiple local windows of fixed size. The pixel features within the low-dynamic reference feature window are used as the query vector, and the pixel features within the corresponding window of the modulation enhancement feature are used as the key and value vectors. By calculating the dot product between the query vector and the key vector and normalizing the result, the attention score is obtained. Finally, the value vector is weighted summed using the attention score to obtain an enhanced representation of the reference feature.

[0070] S3, training a diffusion model for high dynamic range image reconstruction based on the diffusion process;

[0071] The high dynamic range image reconstruction diffusion model is trained, and the training steps are as follows:

[0072] Input high and low resolution image datasets and the total number of diffusion steps T, where y0 is a high dynamic range image, I2 is its corresponding low dynamic range image, and I1 and I3 are adjacent frames of I2 respectively.

[0073] The neural network parameters of the diffusion model for high dynamic range image reconstruction are randomly initialized, and the gradient descent operation is performed according to the loss function. The loss function is as follows:

[0074] ||ε-ε θ (y t ,gθ (I1.I2,I3),t)||ε~N(0,1)

[0075] Where ||·|| represents the L1 paradigm, t randomly selects a number from {1, 2, ..., T}, and N(0,1) is a normal distribution with mean 0 and variance 1. Through gradient backpropagation, the multi-exposure image feature guidance network g based on the spatial attention mechanism can be jointly trained. θ and the conditional noise prediction network ε based on the interactive attention mechanism θ .

[0076] The parameters of the high dynamic range image reconstruction diffusion model are optimized, and the neural network is iteratively updated until the loss function converges, and the training of the high dynamic range image reconstruction diffusion model is completed.

[0077] S4, based on the inverse process of the diffusion model, using the predicted noise to iteratively reconstruct the high dynamic range image;

[0078] The inference process of the diffusion model for high dynamic range image reconstruction can be defined as a reverse Markov process, which is the inverse process of the forward diffusion process. T Start, y T Obeying Gaussian distribution:

[0079] y T ~N(0,1)

[0080] A predicted high dynamic range image with lower noise is obtained through step-by-step reasoning, where the step-by-step reasoning process conforms to the state transition of Markov and obeys the conditional Gaussian distribution:

[0081]

[0082] Among them, p θ (y t-1 |y t ,x) indicates that it contains a learnable conditional Gaussian distribution, which is specifically reflected in μ θ (I1,I2,I3,y t ,γ t ) is the mean of the learnable distribution, is the distribution variance, set to the default value 1-α t .

[0083] Assume that given y t and y0,y t-1 The posterior distribution of is a conditional Gaussian distribution:

[0084]

[0085] By using the Bayesian formula, we can deduce y t-1The mean of the posterior distribution q:

[0086]

[0087] The distribution p that can be learned by a neural network θ Fit the posterior distribution, specifically, estimate the noise ε by reconstructing the diffusion model of the high dynamic range image. Obtain an approximate high dynamic range image by transforming

[0088]

[0089] Thus we get the learnable distribution p θ The mean of is:

[0090]

[0091] The diffusion model for high dynamic range image reconstruction can finally be inferred step by step through reparameter sampling:

[0092]

[0093] The entire inverse process is performed for T steps, where T = 1000, i.e., t = T, T-1, …, 1. In the initial inverse step, the Gaussian noise image, the low dynamic range image, and the time t of the current inverse step are input into the diffusion model for high dynamic range image reconstruction. The inference result is then input into the diffusion model for the next step. After T inverse steps, the final high dynamic range reconstructed image is obtained.

[0094] In summary, the present invention proposes a multi-exposure image high dynamic range image reconstruction method based on a diffusion model, which adopts forward denoising and reverse denoising diffusion processing for high dynamic range images, and simultaneously utilizes effective network guidance strategy and interaction strategy to realize multi-exposure image reconstruction of high dynamic range images.

[0095] Example 2:

[0096] See also Figure 4 , based on Example 1 but different in that,

[0097] The advanced comparison methods selected in this paper include: AHDR, HDRGAN, where AHDR is a method based on L1 loss constraint, and HDRGAN is a method that uses GAN to reconstruct high dynamic range images. For these methods, we tested them on the same dataset and obtained Figure 4 The effect diagram shown.

[0098] from Figure 4 It can be seen that the multi-exposure image high dynamic range image reconstruction method based on the diffusion model proposed in the present invention can effectively alleviate the motion artifact problem and generate semantically consistent details in the over-exposed area.

[0099] Table 1 shows the comparative experimental results of various comparison methods. μ and L represent the calculation indicators in the tone mapping domain and the linear domain, where PSNR is the peak signal-to-noise ratio indicator, SSIM measures the structural similarity, and FID is the perceptual indicator.

[0100] Table 1

[0101]

[0102]

[0103] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A method for reconstructing high dynamic range images from multi-exposure images based on a diffusion model, characterized in that: The following steps are involved: S1, construction of diffusion process of high dynamic range image reconstruction diffusion model; S2. Establish a high dynamic range image reconstruction diffusion model, take the multi-exposure low dynamic range image as input, and output the predicted noise through the high dynamic range image reconstruction diffusion model; S3, training the high dynamic range image reconstruction diffusion model based on the diffusion process constructed in S1; S4, based on the inverse process of the diffusion model, using the predicted noise to iteratively reconstruct the high dynamic range image; The high dynamic range image reconstruction diffusion model includes the following two parts: Multi-exposure image feature guidance network based on spatial attention mechanism g θ :The multi-exposure image feature guide network g θ It is composed of a spatial attention module that takes the reference frame I 2 and adjacent frames I 1 and I 3 as input, first pass through the convolution layer to extract image features respectively, splice the reference frame features and adjacent frame features along the feature channel direction, input multiple cascaded convolution-activation layers to generate a set of spatial weights, and multiply the spatial weights with the adjacent frames to obtain the alignment features of the adjacent frames; The aligned features are then transformed through a linear layer and passed through multiple convolutional layers to generate spatial dynamic affine transformation parameters; the spatial dynamic affine transformation parameters include scaling parameters and offset parameters; finally, the features of the reference frame are modulated using the affine transformation parameters to obtain modulation enhancement features; Conditional noise prediction network based on interactive attention mechanism ε θ : It consists of a U-Net-type network based on pixel-level cross attention. It first downsamples the input features and then restores the original feature size through upsampling. In each layer of the U-Net network, it includes a convolution-window-based pixel-level interactive attention-convolution structure, and adds residual connections in each layer of the network. Among them, the window-based pixel-level interactive attention specifically includes the following: Reference Frame I 2. Low-dynamic reference features are extracted through the convolutional layer. Subsequently, the low-dynamic reference features and modulation enhancement features are divided into multiple local windows of fixed size. The pixel features within the low-dynamic reference feature window are used as the query vector, and the pixel features within the corresponding window of the modulation enhancement feature are used as the key and value vectors. The dot product between the query vector and the key vector is then calculated, and the result is normalized to obtain the attention score. Finally, the attention score is used to perform weighted summation on the value vector to obtain the enhanced representation of the reference feature.

2. The method for reconstructing a high dynamic range image from multiple exposure images based on a diffusion model according to claim 1, wherein: The S1 specifically includes the following contents: Gaussian noise is repeatedly added to the high dynamic range image through a diffusion process, assuming that the noise addition process conforms to the Gaussian distribution: Among them, y t and y t-1 denote the high dynamic range noise images at step t and step t-1 respectively; q(y t |y t-1 ) is y t Towards y t-1 The diffusion process, α t Used to determine the variance of the noise added in each iteration, the value range is 0<α t <1; I represents the identity matrix; According to the superposition of normal distribution, high dynamic range image y 0 spread to y t The intermediate step is edge processing to obtain a high dynamic range image y 0 condition, the t-th step high dynamic range noise image y t The conditional probability distribution of : in, ; According to this distribution, the high dynamic range noise image y is obtained by re-parameter sampling t The expression: in, represents a random noise image sampled from a normal distribution with mean 0 and variance 1; The entire diffusion process will produce high dynamic range images y 0 as input, after a continuous t-step diffusion process, the output is a standard Gaussian noise image y T ; Where t = 1, 2, …, T, T represents the total number of input high- and low-resolution image datasets and diffusion steps, T = 1000.

3. The method for reconstructing a high dynamic range image from multiple exposure images based on a diffusion model according to claim 2, wherein: The S3 specifically includes the following contents: The neural network parameters of the diffusion model for high dynamic range image reconstruction are randomly initialized, and a gradient descent operation is performed according to the loss function, wherein the loss function is specifically: in, represents the L1 paradigm; t represents any random number in {1,2,…,T}; N(0,1) represents a normal distribution with mean 0 and variance 1; express ε Obeying N(0,1), through gradient back propagation, jointly train the multi-exposure image feature guidance network based on the spatial attention mechanism g θ and the conditional noise prediction network ε based on the interactive attention mechanism θ ; I 2 represents the reference frame; I 1. I 3 indicates adjacent frames; The parameters of the high dynamic range image reconstruction diffusion model are optimized, and the neural network is iteratively updated until the loss function converges, and the training of the high dynamic range image reconstruction diffusion model is completed.

4. The method for reconstructing a high dynamic range image from multiple exposure images based on a diffusion model according to claim 3, wherein: The S4 specifically includes the following contents: The inference process of the diffusion model for high dynamic range image reconstruction is defined as a reverse Markov process, which is the inverse process of the forward diffusion process. y T start, y T Obeying Gaussian distribution: y T ~ N (0,1) A predicted high dynamic range image with lower noise is obtained through step-by-step reasoning, where the step-by-step reasoning process conforms to the state transition of Markov and obeys the conditional Gaussian distribution: in, I represents the identity matrix; Represents a learnable conditional Gaussian distribution; represents the mean of the learnable distribution; Represents the distribution variance, which is set to the default value 1-α t ; Assume that given y t and y0,y t-1 The posterior distribution of is a conditional Gaussian distribution: Derived by Bayesian formula t-1 The mean of the posterior distribution q: The distribution p learned by the neural network θ Fit the posterior distribution, estimate the noise ε through the high dynamic range image reconstruction diffusion model, and obtain an approximate high dynamic range image through transformation : Then we get p θ The mean of is: The diffusion model for high dynamic range image reconstruction ultimately performs step-by-step inference via reparameter sampling: in, The entire inverse process is performed for T steps, T=1000, that is, t=T,T-1,…,1; the initial inverse step inputs the Gaussian noise image, the low dynamic range image, and the time t of the current inverse step into the high dynamic range image reconstruction diffusion model, and the inference result is used as the diffusion model input for the next step. After T inverse steps, the final high dynamic range reconstructed image is obtained.

Images