A hdr imaging method based on bayesian neural network

Abstract

The application discloses an HDR imaging method based on a Bayesian neural network, which comprises the following steps: firstly, mapping input LDR images with different exposures to a linear domain; secondly, connecting the LDR images with different exposures and the images in the linear domain, and inputting the images into a feature extractor to obtain features; thirdly, obtaining new features through a convolution attention module and a convolution layer; and finally, obtaining an HDR image through a feature fusion module. The application can reduce network parameters, expand a receptive field, make network calculation faster, and improve performance.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to an HDR imaging method. Background Technology

[0002] The goal of high dynamic range (HDR) imaging is to estimate a potential high-quality image from a multi-exposure low dynamic range (LDR) input. Intuitively, there are various possible HDR images corresponding to a given LDR input, which leads to uncertainty in the estimation results. However, most existing HDR imaging methods only use l1 or l2 loss to provide a possible estimate from various possible solutions, failing to model the uncertainty and thus lacking high-frequency details. The literature "An Improved Pyramid Multi-Exposure HDR Image Generation Method, Modern Computer, 2020, Vol 27(15), pp 130-136" discloses a multi-exposure HDR image generation method based on an improved pyramid. This method addresses the imaging problems such as color shift and detail blurring in the current RGB space generation of high dynamic range images by designing a new HDR imaging system. By extracting the salient features of the denoised multi-exposure LDR image sequence and assigning different weight parameters according to the salient features of the image, the three feature factors of contrast, color saturation, and exposure brightness are fused into the improved pyramid model to complete the HDR image generation. The method described in the literature generates an HDR image based on a multi-exposure LDR, without addressing the uncertainty of generating different HDR images, and cannot provide the reliability of the generated HDR image. Furthermore, it is unable to model uncertainty and lacks high-frequency detailed information. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention provides an HDR imaging method based on a Bayesian neural network. First, input LDR images with different exposures are mapped to a linear domain. Then, the LDR images with different exposures are concatenated with the images in the linear domain and input into a feature extractor to obtain features. Next, new features are obtained through a convolutional attention module and convolutional layers. Finally, an HDR image is obtained through a feature fusion module. This invention can reduce network parameters, expand the receptive field, and make network computation faster and performance better.

[0004] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0005] Step 1: Map the input LDR images L1, L2, L3 with different exposures into the linear domain to obtain H1, H2, H3;

[0006] Step 2: Connect the LDR images L1, L2, L3 with different exposures to H1, H2, H3 in the linear domain to obtain X1, X2, X3;

[0007] Step 3: Place X i Input the feature extractor to obtain feature F i ;

[0008] Step 4: Feature F i The feature F′ is obtained after passing through the convolutional attention module and convolutional layer. i ;

[0009] Step 5: Feature F′ i The HDR image is obtained after feature fusion module.

[0010] Preferably, step 1 specifically comprises:

[0011] Using gamma correction as the mapping function, the mapping process is as follows:

[0012]

[0013] In the formula, t i Let L1 represent the exposure time of the i-th image, and γ be the gamma correction parameter.

[0014] Preferably, step 2 specifically comprises:

[0015] The LDR images with different exposures and the images mapped to the linear domain are concatenated along the channel dimension into a 6-channel input, as shown below:

[0016] X i ={L i H i}, i = 1, 2, 3 (2)

[0017] Preferably, the feature extractor in step 3 is:

[0018] In the feature extractor, convolutional layers and variational Bayesian layers are used to capture fixed and variable features;

[0019] Introducing variational Bayesian layers w V Learning from input X i i=1,2,3 to the target The one-to-many capability; using a hierarchical prior with a zero-mean Gaussian distribution, and following a uniform super-prior, i.e., the sampling variance σ in U(a,b), as shown below:

[0020] p(w V )=N(0,σ), p(σ)=U(a,b) (3)

[0021] The process of learning the variability of intermediate features is as follows:

[0022]

[0023] In the formula, w V These are network parameters. It is the input to the variational Bayesian layer, and ⊙ represents the Hadamard product.

[0024] Preferably, step 4 specifically comprises:

[0025] Using the intermediate exposure image L2 as the reference image, the features F1 and F3 of the non-reference image are input together with the feature map F2 of the reference image into the convolutional attention module a. j (·), j = 1, 3, and then obtain attention map A. j The operation is as follows:

[0026] A j =a j (F j ,F2) (5)

[0027] The predicted attention map then focuses on the features of the non-reference image through point-by-point multiplication to obtain new non-reference image features. As shown below:

[0028]

[0029] New non-reference image features The new feature F′ is obtained by passing the reference image feature F2 through a convolutional layer. i .

[0030] Preferably, step 5 specifically comprises:

[0031] First, the feature F′ i The features are superimposed and merged to obtain feature F″.

[0032] The feature fusion network consists of multiple convolutional layers, dynamic receptive field layers, and variational Bayesian layers. F″ first passes through three convolutional layers, then three dynamic receptive field layers and variational Bayesian layers, and finally a single convolutional layer generates the final HDR image.

[0033] Since variational Bayesian layers require more parameters and cannot increase the receptive field of the network, this invention designs a dynamic receptive field layer with fewer parameters, and to reduce complexity, the posterior distribution of the parameters in the dynamic receptive field layer is defined as uniform distribution.

[0034] The dynamic receptive field layer consists of three branches. The first branch consists of a 1×1 dynamic convolution and a 3×3 dilated convolution with a dilation rate of 1. The second branch consists of a 1×1 and a 3×3 dynamic convolution and a 3×3 dilated convolution with a dilation rate of 2. The third branch consists of a 1×1 and two 3×3 dynamic convolutions and a 3×3 dilated convolution with a dilation rate of 3. The outputs of these three branches are concatenated and passed through a 1×1 dynamic convolution. Then, the result is added to the input after the 1×1 dynamic convolution to obtain the output of the dynamic receptive field layer.

[0035] The dynamic convolution first obtains the coefficients of the four convolution kernels through the attention module, then multiplies the obtained coefficients with the input convolution, and finally adds them to the input convolution.

[0036] Using B0 as the input to the dynamic convolution module and B1 as the output, the attention module is calculated as follows:

[0037] B T =ReLU(FC1(AvgPool(B0))) (7)

[0038]

[0039] In the formula, AvgPool(·) is the average pooling operation, FC1 and FC2 represent fully connected networks, and four coefficients are obtained through the attention module.

[0040] The final output B1 of the dynamic convolution module is calculated by equation (9):

[0041]

[0042] In the formula, Conv(·) represents a 1×1 convolutional layer.

[0043] The beneficial effects of this invention are as follows:

[0044] This invention uses convolutional layers and variational Bayesian layers to capture fixed and variable features. Variational Bayesian layers can model one-to-many relationships and provide various possible solutions. Utilizing the Bayesian framework, a variety of potentially high-quality estimates can be provided, especially in high-frequency details. HDR reconstruction is then performed through a feature fusion module. To reduce complexity and expand the receptive field, this invention uses a dynamic receptive field layer in the feature fusion module, reducing network parameters, expanding the receptive field, and making network computation faster and performance better. Attached Figure Description

[0045] Figure 1 This is a flowchart of the HDR imaging based on Bayesian neural networks of the present invention.

[0046] Figure 2 The images shown are three LDR images L1, L2, and L3 with different exposures, representing an embodiment of the present invention.

[0047] Figure 3 The HDR image finally reconstructed in this embodiment of the invention Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0049] like Figure 1 As shown, an HDR imaging method based on a Bayesian neural network includes the following steps:

[0050] Step 1: Map the input LDR images L1, L2, and L3 with different exposures into the linear domain to obtain H1, H2, and H3.

[0051] Linearizing the input LDR images with different exposures into the original format of HDR imaging is crucial. Since gamma correction essentially maps the input image to a domain closer to human perception, this invention uses it as the mapping function, and the mapping process is as follows:

[0052]

[0053] In the formula, t i Let L1 represent the exposure time of the i-th image, and γ be the gamma correction parameter.

[0054] Step 2: Connect the LDR images L1, L2, L3 with different exposures to H1, H2, H3 in the linear domain to obtain X1, X2, X3.

[0055] The LDR images with different exposures and the images mapped to the linear domain are concatenated along the channel dimension into a 6-channel input, as shown below:

[0056] X i ={L i H i}, i = 1, 2, 3 (2)

[0057] Step 3: Place X i Input the feature extractor to obtain feature F i , .

[0058] In the feature extractor, this invention uses convolutional layers and variational Bayesian layers to capture both fixed and variable features. To understand the fundamental one-to-many property in HDR image restoration and estimate uncertainty, this invention introduces variational Bayesian layers. This invention does not use convolutional layers to learn a one-to-one mapping between LDR and HDR images, i.e., it does not fix the parameters w after training. CThis is because, in reality, the mapping from high-quality scenes to distorted images is surjective. For example, an HDR image might correspond to the distorted results of multiple natural images. Therefore, restoring or approximating the local image manifold is more efficient than simply fitting a true HDR image. More intuitive. This invention introduces a variational Bayesian layer w. V Let's learn from input X i i=1,2,3 to the target One-to-many capability. In order to... V To constrain variability, this invention uses a hierarchical prior with a zero-mean Gaussian distribution and follows a uniform super-prior (i.e., U(a,b)) with a sampling variance σ, as shown below:

[0059] p(w V )=N(0,σ), p(σ)=U(a,b) (3)

[0060] This invention uses these parameters to learn the variability of intermediate features, as follows:

[0061]

[0062] In the formula, w V These are network parameters. This is the input to the variational Bayesian layer, and ⊙ represents the Hadamard product. For CNN-based networks, given data, the posterior distribution of network parameters is uniform, which tends to generate results similar to the true values. Since the loss function considers the same error for different results, the final result often produces blur artifacts. Compared to CNNs, this invention introduces a variational Bayesian layer to learn different networks for different results in natural image manifolds, predicting images with clearer details. Furthermore, since different results can be obtained using the variational Bayesian layer, this invention can estimate the uncertainty of the proposed network.

[0063] Step 4: Feature F i The feature F′ is obtained after passing through the convolutional attention module and convolutional layer. i .

[0064] This invention uses the medium-exposure image as the reference image. First, the features F1 and F3 of the non-reference image are input together with the feature map F2 of the reference image into the convolutional attention module a. i (·), i = 1, 3, then obtain attention map A i For i = 1, 3, the operation is as follows:

[0065] A i =a i (F i ,F2),i=1,3 (5)

[0066] The predicted attention map then focuses on the features of the non-reference image through point-by-point multiplication to obtain new non-reference image features. As shown below:

[0067]

[0068] New non-reference image features The new feature F′ is obtained by passing the reference image feature F2 through a convolutional layer. i , where i = 1, 2, 3.

[0069] Step 5: Feature F′ i HDR images are obtained after feature fusion module.

[0070] In the designed feature fusion network, this invention first integrates feature F′ i The features are superimposed and merged to obtain feature F″. The feature fusion network consists of multiple convolutional layers, dynamic receptive field layers, and variational Bayesian layers. F″ first passes through three convolutional layers, then through three dynamic receptive field layers and variational Bayesian layers, and finally generates the final HDR image by a single convolutional layer.

[0071] Since variational Bayes layers require more parameters and cannot increase the receptive field of the network, this invention designs a dynamic receptive field layer with fewer parameters. To reduce complexity, the posterior distribution of the parameters in the dynamic receptive field layer is defined as uniform. The dynamic receptive field layer consists of multiple branches: the first branch consists of a 1×1 dynamic convolution and a 3×3 dilated convolution with a dilation rate of 1; the second branch consists of a 1×1 and a 3×3 dynamic convolution and a 3×3 dilated convolution with a dilation rate of 2; the third branch consists of a 1×1 and two 3×3 dynamic convolutions and a 3×3 dilated convolution with a dilation rate of 3. The outputs of these three branches are concatenated and passed through a 1×1 dynamic convolution, then added to the input after the 1×1 dynamic convolution to obtain the output of the dynamic receptive field layer.

[0072] Dynamic convolution does not use the fixed parameters of a CNN; instead, it adaptively aggregates four convolutional kernels through attention to improve model performance. First, the coefficients of the four convolutional kernels are obtained through an attention module. Then, these coefficients are multiplied by the input convolution, and finally added back to the input convolution. The following section describes the operation of dynamic convolution using B0 as the input and B1 as the output, with the attention module calculated as follows:

[0073] B T =ReLU(FC1(AvgPool(B0))) (7)

[0074]

[0075] In the formula, AvgPool(·) is the average pooling operation, FC1 and FC2 represent fully connected networks, and four coefficients are obtained through the attention module. The final output B1 of the dynamic convolution module is calculated using the following formula:

[0076]

[0077] In the formula, Conv(·) represents a 1×1 convolutional layer. Specific implementation examples:

[0079] The following example uses three LDR images L1, L2, and L3 with different exposures. Figure 2 The specific implementation method is described below. This embodiment of the HDR imaging method based on a Bayesian neural network includes the following steps:

[0080] Step 1: Map the input LDR images L1, L2, and L3 with different exposures into the linear domain to obtain H1, H2, and H3.

[0081] Linearizing the input LDR images with different exposures into the original format of HDR imaging is crucial. Since gamma correction essentially maps the input image to a domain closer to human perception, this invention uses it as the mapping function, and the mapping process is as follows:

[0082]

[0083] In the formula, t i Let L1 represent the exposure time of the i-th image, and γ be the gamma correction parameter.

[0084] Step 2: Connect the LDR images L1, L2, L3 with different exposures to H1, H2, H3 in the linear domain to obtain X1, X2, X3.

[0085] The LDR images with different exposures and the images mapped to the linear domain are concatenated along the channel dimension into a 6-channel input, as shown below:

[0086] X i ={L i H i}, i = 1, 2, 3 (2)

[0087] Step 3: Place X i Input the feature extractor to obtain feature F i .

[0088] In the feature extractor, this invention uses convolutional layers and variational Bayesian layers to capture both fixed and variable features. To understand the fundamental one-to-many property in HDR image restoration and estimate uncertainty, this invention introduces variational Bayesian layers. This invention does not use convolutional layers to learn a one-to-one mapping between LDR and HDR images, i.e., it does not fix the parameters w after training. C This is because, in reality, the mapping from high-quality scenes to distorted images is surjective. For example, an HDR image might correspond to the distorted results of multiple natural images. Therefore, restoring or approximating the local image manifold is more efficient than simply fitting a true HDR image. More intuitive. This invention introduces a variational Bayesian layer w. V Let's learn from input X i i=1,2,3 to the target One-to-many capability. In order to... V To constrain variability, this invention uses a hierarchical prior with a zero-mean Gaussian distribution and follows a uniform super-prior (i.e., U(a,b)) with a sampling variance σ, as shown below:

[0089] p(w V )=N(0,σ), p(σ)=U(a,b) (3)

[0090] This invention uses these parameters to learn the variability of intermediate features, as follows:

[0091]

[0092] In the formula, w V These are network parameters. This is the input to the variational Bayesian layer, and ⊙ represents the Hadamard product. For CNN-based networks, given data, the posterior distribution of network parameters is uniform, which tends to generate results similar to the true values. Since the loss function considers the same error for different results, the final result often produces blur artifacts. Compared to CNNs, this invention introduces a variational Bayesian layer to learn different networks for different results in natural image manifolds, predicting images with clearer details. Furthermore, since different results can be obtained using the variational Bayesian layer, this invention can estimate the uncertainty of the proposed network.

[0093] Step 4: Feature F i The feature F′ is obtained after passing through the convolutional attention module and convolutional layer. i .

[0094] This invention uses the medium-exposure image as the reference image. First, the features F1 and F3 of the non-reference image are input together with the feature map F2 of the reference image into the convolutional attention module a. i (·), i = 1, 3, then obtain attention map Ai For i = 1, 3, the operation is as follows:

[0095] A i =a i (F i ,F2),i=1,3 (5)

[0096] The predicted attention map then focuses on the features of the non-reference image through point-by-point multiplication to obtain new non-reference image features. As shown below:

[0097]

[0098] New non-reference image features The new feature F′ is obtained by passing the reference image feature F2 through a convolutional layer. i , where i = 1, 2, 3.

[0099] Step 5: Feature F′ i HDR images are obtained after feature fusion module. like Figure 3 As shown.

[0100] In the designed feature fusion network, this invention first integrates feature F′ i The features are superimposed and merged to obtain feature F″. The feature fusion network consists of multiple convolutional layers, dynamic receptive field layers, and variational Bayesian layers. F″ first passes through three convolutional layers, then through three dynamic receptive field layers and variational Bayesian layers, and finally generates the final HDR image by a single convolutional layer.

[0101] Since variational Bayes layers require more parameters and cannot increase the receptive field of the network, this invention designs a dynamic receptive field layer with fewer parameters. To reduce complexity, the posterior distribution of the parameters in the dynamic receptive field layer is defined as uniform. The dynamic receptive field layer consists of multiple branches: the first branch consists of a 1×1 dynamic convolution and a 3×3 dilated convolution with a dilation rate of 1; the second branch consists of a 1×1 and a 3×3 dynamic convolution and a 3×3 dilated convolution with a dilation rate of 2; the third branch consists of a 1×1 and two 3×3 dynamic convolutions and a 3×3 dilated convolution with a dilation rate of 3. The outputs of these three branches are concatenated and passed through a 1×1 dynamic convolution, then added to the input after the 1×1 dynamic convolution to obtain the output of the dynamic receptive field layer.

[0102] Dynamic convolution does not use the fixed parameters of a CNN; instead, it adaptively aggregates four convolutional kernels through attention to improve model performance. First, the coefficients of the four convolutional kernels are obtained through an attention module. Then, these coefficients are multiplied by the input convolution, and finally added back to the input convolution. The following section describes the operation of dynamic convolution using B0 as the input and B1 as the output, with the attention module calculated as follows:

[0103] B T =ReLU(FC1(AvgPool(B0))) (7)

[0104]

[0105] In the formula, AvgPool(·) is the average pooling operation, FC1 and FC2 represent fully connected networks, and four coefficients are obtained through the attention module. The final output B1 of the dynamic convolution module is calculated using the following formula:

[0106]

[0107] In the formula, Conv(·) represents a 1×1 convolutional layer.

Claims

1. A Bayesian neural network-based HDR imaging method, characterized in that, Includes the following steps: Step 1: Input LDR images with different exposures , , Mapping to a linear domain, we get , , ; Step 2: Connect LDR images with different exposures 、 、 With linear domain , , get , , ; Step 3: Put Input the feature extractor to obtain features ; Step 4: Features Features are obtained after passing through a convolutional attention module and a convolutional layer. ; Step 5: Features The HDR image is obtained after feature fusion module. ; The feature extractor in step 3 is: In the feature extractor, convolutional layers and variational Bayesian layers are used to capture fixed and variable features; Introduction Variational Bayesian layer learning from input To the target The one-to-many capability; using a hierarchical prior with a zero-mean Gaussian distribution, and following a uniform over-prior, i.e. Sampling variance As shown below: (3) The process of learning the variability of intermediate features is as follows: (4) In the formula, These are network parameters. It is the input of the variational Bayesian layer. Represents the Hadamard product; Step 5: Features The HDR image is obtained after feature fusion module. for: First, the features By superimposing and merging these features, we can obtain the characteristics. ; The feature fusion module consists of multiple convolutional layers, dynamic receptive field layers, and variational Bayesian layers. First, the image passes through three convolutional layers, then three dynamic receptive field layers and one variational Bayesian layer, and finally a single convolutional layer generates the final HDR image. ; The posterior distribution of parameters in the dynamic receptive field layer is defined as uniform distribution; The dynamic receptive field layer consists of three branches. The first branch consists of a 1×1 dynamic convolution and a 3×3 dilated convolution with a dilation rate of 1. The second branch consists of a 1×1 and a 3×3 dynamic convolution and a 3×3 dilated convolution with a dilation rate of 2. The third branch consists of a 1×1 and two 3×3 dynamic convolutions and a 3×3 dilated convolution with a dilation rate of 3. The outputs of these three branches are concatenated and then passed through a 1×1 dynamic convolution. The dynamic convolution first obtains the coefficients of the four convolution kernels through the attention module, then multiplies the obtained coefficients with the input convolution, and finally adds them to the input convolution. by As input to the dynamic convolution module, As output, the attention module is calculated as follows: (7) (8) In the formula It is an average pooling operation. and This represents a fully connected network, with four coefficients obtained through the attention module. ; The final output of the dynamic convolution module Calculated from equation (9): (9) In the formula This represents a 1×1 convolutional layer.

2. The HDR imaging method based on a Bayesian neural network according to claim 1, characterized in that, Step 1 specifically involves: Using gamma correction as the mapping function, the mapping process is as follows: (1) In the formula, Indicates the first One image Exposure time, It is the gamma correction parameter.

3. The HDR imaging method based on a Bayesian neural network according to claim 1, characterized in that, Step 2 specifically involves: The LDR images with different exposures and the images mapped to the linear domain are concatenated along the channel dimension into a 6-channel input, as shown below: (2)。 4. The HDR imaging method based on a Bayesian neural network according to claim 1, characterized in that, Step 4 specifically involves: Intermediate exposure image As a reference image, the features of the non-reference image are respectively... and Feature map of reference image Together Input Convolutional Attention Module Then obtain the attention map. The operation is as follows: (5) The predicted attention map then focuses on the features of the non-reference image through point-by-point multiplication to obtain new non-reference image features. As shown below: (6) New non-reference image features , Features of the reference image New features are obtained after convolutional layers. .

Images