A dynamic scene HDR imaging method based on LSGAN
Patent Information
- Application Number
- CN202310755809.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-25
- Publication Date
- 2026-10-30
- Estimated Expiration
- 2043-06-25
AI Technical Summary
[0005]本发明的目的是克服上述现有技术的缺陷,提供一种基于LSGAN的动态场景HDR成像方法,通过最小二乘生成对抗网络(LSGAN)模型来实现动态场景下的HDR成像,通过两阶段训练使生成的图像在结构内容和数据分布上都接近真实的HDR图像,生成器利用全局膨胀残差密集连接模块(Global Dilated Residual Dense Block,GDRDB)解决了网络因感受野受限而无法充分利用全局信息的问题,有效地提升了网络恢复图像结构信息、去除鬼影和提高图像整体视觉质量的能力
[0041]本发明所述方法实现了动态场景下的无鬼影HDR成像;(1)生成器中引入GDRDB能有解决鬼影问题,特征融合网络由多个GDRDB组成,为恢复图像信息和鉴别鬼影特征提供了充分的局部信息和全局上下文信息,帮助网络恢复图像的结构、纹理等高频信息,以及帮助网络找出破坏图像一致性的特征并修复这些不自然的区域,提高网络还原信息和去鬼影的能力,解决了现有的基于深度神经网络的算法感受野受限的问题,具有较强的鉴别鬼影特征和信息恢复的能力。(2)生成器使用全局残差学习策略,提高了网络的学习效率,全局残差学习让网络只需要学习参考图像的残差特征,能减轻网络的学习负担,避免过拟合。(3)使用了内容损失和最小二乘对抗损失组成的损失函数训练网络,使得生成器不仅能生成在结构上逼近真实图像的HDR图像,同时能使生成的图像在数据分布上也与真实的HDR图像相近,视觉上可以表现为更少的鬼影和更丰富的图像细节,图像具有更高的感知质量。
Smart Images

Figure CN116801111B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, specifically relating to a dynamic scene HDR imaging method based on LSGAN. Background Technology
[0002] With the rapid development of science and technology and digital imaging technology, the demand for High Dynamic Range (HDR) imaging technology in daily life and scientific research is constantly increasing. Multi-exposure HDR imaging technology fuses information from multiple frames of Low Dynamic Range (LDR) images to generate HDR images that closely resemble the real scene. Dynamic scenes generally refer to scenes where there may be moving objects, camera shake, or other factors during shooting. In such cases, there is motion displacement between the multiple frames of LDR images. Multi-exposure HDR imaging in these situations faces many challenges, including artifacts and information recovery issues. Therefore, research on dynamic scene HDR imaging technology has received widespread attention from researchers.
[0003] Dynamic scene HDR imaging methods can be divided into traditional methods and deep learning-based methods. Traditional dynamic scene HDR imaging methods often employ moving pixel culling, probabilistic optimization, or alignment-based fusion. While these algorithms can generate ghost-free HDR images with few moving pixels, they fail to effectively recover artifacts and image details when the object's motion is significant or the exposure in the reference image is complex. With the development of deep learning technology, convolutional neural networks (CNNs) learn the mapping relationship between the network's input and output through multi-layer nonlinear transformations, enabling various complex transformations and demonstrating powerful performance in data fitting and feature extraction.
[0004] In recent years, deep learning technology has achieved some success in addressing the ghosting and information recovery problems in dynamic scene HDR imaging. Current deep learning-based dynamic scene HDR imaging models primarily fuse three LDR images with different exposures into a single HDR image, using an image with a moderate exposure time as a reference image. The final HDR image is structurally aligned with the reference image, containing all information from both bright and dark areas of the scene. Deep learning-based dynamic scene HDR imaging methods, leveraging attention mechanisms, residual modules, and generative adversarial networks, effectively overcome the shortcomings of traditional methods, achieving ghosting-free alignment and fusion in scenes with large foreground motion. However, when the input LDR image sequence is severely overexposed / underexposed, the convolutional neural network is limited by its receptive field, only able to extract limited local features, resulting in poor-quality HDR images. Improving the network's receptive field to simultaneously extract local and global features from multiple frames, and utilizing these extracted features to suppress ghosting and recover scene information, remains a key technical challenge. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a dynamic scene HDR imaging method based on LSGAN. It realizes HDR imaging in dynamic scenes through the least squares generative adversarial network (LSGAN) model. Through two-stage training, the generated images are close to real HDR images in terms of structural content and data distribution. The generator uses the Global Dilated Residual Dense Block (GDRDB) module to solve the problem that the network cannot fully utilize global information due to the limited receptive field, effectively improving the network's ability to recover image structural information, remove ghosting, and improve the overall visual quality of the image.
[0006] The technical problem addressed by this invention is solved as follows:
[0007] A dynamic scene HDR imaging method based on LSGAN includes the following steps:
[0008] Step 1. Construction of Generative Adversarial Networks
[0009] Generative adversarial networks consist of a generator and a decision maker;
[0010] The generator consists of an encoder, a feature fusion network, and a decoder, which are cascaded in sequence.
[0011] The encoder includes three shallow feature extraction network branches, a first convolutional layer, and a second convolutional layer. Three LDR images with different exposure times are input into the three shallow feature extraction network branches respectively to obtain the shallow features of the three LDR images. The shallow features of the three LDR images are merged by channel and then input into the first convolutional layer and the second convolutional layer in sequence to perform preliminary fusion of the shallow features of the three LDR images.
[0012] The feature fusion network consists of several cascaded global dilated residual dense connection modules. Each global dilated residual dense connection module consists of cascaded dilated residual dense connection modules and a global branch. Each dilated residual dense connection module includes multiple cascaded dilated convolutional layers and a fifth convolutional layer, which further extract and fuse the initially fused image features. The global branch includes a cascaded global average pooling layer and a sixth convolutional layer, which are used to extract the global vector of the output features of the dilated residual dense connection modules and output it after merging with the output features of the dilated residual dense connection modules.
[0013] The decoder consists of six convolutional layers and one global residual connection layer. After the first three convolutional layers, bilinear upsampling is cascaded to restore the image to its original size. Then, after two cascaded convolutional layers, the residual image is extracted. The LDR image with the middle exposure time is used as the reference image. The residual image and the reference image are input to the global residual connection layer, added pixel by pixel, and then input to the last convolutional layer to obtain the final generated HDR image.
[0014] The decision maker consists of four convolutional layers and one fully connected layer cascaded in sequence. The kernels of the four convolutional layers are all 3×3. The first three convolutional layers introduce spectral normalization (SN), and the last three convolutional layers use batch normalization (BN). The activation function of the first three convolutional layers is ReLU.
[0015] Step 2. Preparation of training data
[0016] Three LDR images with different exposure times and their gamma-transformed images are merged by channel, randomly cropped into image blocks of size 256×256, and used as training data. The pixel values are adjusted to the range of [0, 1].
[0017] Step 3. Pre-train the generator
[0018] The training data is input into the generator, and the content loss function L is selected as the loss function. con :
[0019] L con =|T(H gt )-T(H pre )|1
[0020] Where T is the tone mapping function, H gt For true HDR images, H pre For the HDR image generated by the generator, ||1 indicates the L1 norm;
[0021] The tone mapping function T is expressed as:
[0022]
[0023] Where H is the image independent variable and μ is the compression coefficient;
[0024] Step 4. Perform adversarial training on the generator and the decision-maker.
[0025] The training data is input into the generator, and the HDR image generated by the generator and the training data are respectively input into the decision unit;
[0026] The generator's adversarial loss function L LSGAN (G) is:
[0027] L LSGAN(G)=E(x)[(D(G(x))-1) 2 ]
[0028] Where x is the training data, E represents the expected value, G(x) represents the HDR image generated by the generator from the training data x, and D(G(x)) represents the output result of the HDR image generated by the generator being input into the discriminator;
[0029] The loss function for training the generator is ultimately expressed as L(G):
[0030] L(G)=L con +αL LSGAN (G)
[0031] Where α is a weight hyperparameter used to balance generative adversarial loss and content loss;
[0032] The loss function for training the discriminator is:
[0033] L(D) = 0.5 × (E(x)[D(G(x))) 2 ]+E(y)[(D(y)-1) 2 ])
[0034] Where y is the real HDR image corresponding to the training data x, and D(y) is the output result of the real HDR image y input to the decision maker;
[0035] Step 5. Input the three real-time LDR images with different exposure times into the trained generator to obtain the final real-time HDR image.
[0036] Furthermore, the shallow feature extraction network branch includes a third convolutional layer, an attention module, and a fourth convolutional layer cascaded in sequence.
[0037] Furthermore, in the encoder, the convolution kernels of the first, second, third, and fourth convolutional layers are 3×3.
[0038] Furthermore, in the feature fusion network, the kernels of the dilated convolutional layers are 3×3 with an inflation factor of 2; the kernels of the fifth and sixth convolutional layers are 1×1; and each dilated convolutional layer is activated by the Leaky ReLU function.
[0039] Furthermore, the convolutional kernels of the six convolutional layers in the decoder and the four convolutional layers in the decision layer are both 3×3.
[0040] The beneficial effects of this invention are:
[0041] The method described in this invention achieves ghost-free HDR imaging in dynamic scenes; (1) The introduction of GDRDB in the generator can solve the ghost problem. The feature fusion network is composed of multiple GDRDBs, which provides sufficient local and global context information for restoring image information and identifying ghost features. It helps the network restore high-frequency information such as the structure and texture of the image, and helps the network find features that destroy the consistency of the image and repair these unnatural areas, thereby improving the network's ability to restore information and remove ghosts. It solves the problem of limited receptive field of existing deep neural network-based algorithms and has a strong ability to identify ghost features and restore information. (2) The generator uses a global residual learning strategy, which improves the learning efficiency of the network. Global residual learning allows the network to learn only the residual features of the reference image, which can reduce the learning burden of the network and avoid overfitting. (3) The network is trained using a loss function composed of content loss and least squares adversarial loss, which enables the generator to not only generate HDR images that are structurally close to real images, but also to make the generated images similar to real HDR images in terms of data distribution. Visually, it can be represented by fewer ghosts and richer image details, and the image has higher perceptual quality. Attached Figure Description
[0042] Figure 1 This is a schematic diagram of the generator structure in the generative adversarial network of the method described in this invention;
[0043] Figure 2 This is a schematic diagram of the Global Dilatation Residual Dense Connection Module (GDRDB) in the method described in this invention.
[0044] Figure 3 This is a schematic diagram of the decision device in the method described in this invention;
[0045] Figure 4 This is a schematic diagram of the training process of the method described in this invention;
[0046] Figure 5 A comparison image on PMAP is generated by the method described in this invention and by existing technologies.
[0047] Figure 6 This is a subjective visual comparison of the generated images on the Kalantari dataset between the method described in this invention and existing technologies.
[0048] Figure 7 This image shows a subjective visual comparison of the generated images on the Tursun dataset between the method described in this invention and existing technologies.
[0049] Figure 8 This is a subjective visual comparison of the generated images on the Sen dataset using the method described in this invention and existing technologies. Detailed Implementation
[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0051] This embodiment provides a dynamic scene HDR imaging method based on LSGAN, including the following steps:
[0052] Step 1. Construction of Generative Adversarial Networks
[0053] Generative adversarial networks consist of a generator and a decision maker;
[0054] The structural diagram of the generator is as follows: Figure 1 As shown, it includes an encoder, a feature fusion network, and a decoder cascaded in sequence; the process of generating an HDR image is divided into the following stages: shallow feature extraction, feature fusion, generation of residual map, and synthesis of HDR image.
[0055] The encoder includes three shallow feature extraction network branches, a first convolutional layer, and a second convolutional layer. The shallow feature extraction network branches include a third convolutional layer, an attention module, and a fourth convolutional layer cascaded in sequence to suppress features in poorly exposed areas and retain effective features in the input image. Three LDR images with different exposure times are input to the three shallow feature extraction network branches respectively to obtain shallow features of the three LDR images. The shallow features of the three LDR images are merged by channel and input to the first and second convolutional layers in sequence to perform preliminary fusion of the shallow features of the three LDR images.
[0056] In the encoder, the convolution kernels of the first, second, third, and fourth convolutional layers are 3×3.
[0057] The shallow features contain information from three LDR images, but contain a lot of redundant information. The feature fusion network uses a Globally Dilated Residual Dense Connection Module (GDRDB) to further fuse the shallow features and extract features that are beneficial for image recovery. These features are then effectively fused to reduce ghosting generated during feature merging. The feature fusion network consists of several cascaded Globally Dilated Residual Dense Connection Modules (GDRDB); a schematic diagram of the GDRDB structure is shown below. Figure 2 As shown, the diagram shows the cascaded dilated residual dense connection module (DRDB) and the global branch (Globao Branch). The DRDB consists of multiple cascaded dilated convolutional layers and a fifth convolutional layer, which further extract and fuse the initially fused image features. The global branch consists of a cascaded global average pooling layer and a sixth convolutional layer, which are used to extract the global vector of the DRDB output features, and then merge it with the DRDB output features before outputting the vector.
[0058] In the feature fusion network, the kernel of the dilated convolutional layer is 3×3 and the dilation coefficient is 2; the kernels of the fifth and sixth convolutional layers are 1×1; each dilated convolutional layer is activated by the Leaky ReLU function, and the fifth convolutional layer plays the role of reducing the number of channels and fusing features.
[0059] The decoder consists of six convolutional layers and one global residual connection layer. The kernels of the six convolutional layers are all 3×3. After the first three convolutional layers, bilinear upsampling is cascaded to restore the image to its original size. Then, after two cascaded convolutional layers, the residual image is extracted. The residual image can fill the gap between the reference image and the real HDR image, expand the dynamic range and enrich the image details. The LDR image with the middle exposure time is used as the reference image. The residual image and the reference image are input to the global residual connection layer, added pixel by pixel, and then input to the last convolutional layer to obtain the final generated HDR image.
[0060] The structural diagram of the decision device is shown below. Figure 3 As shown, it includes four convolutional layers and one fully connected layer in sequence. The convolutional kernels of the four convolutional layers are all 3×3. The first three convolutional layers introduce spectral normalization (SN), and the last three convolutional layers use batch normalization (BN). The activation function of the first three convolutional layers is ReLU.
[0061] In this embodiment, the generator generates HDR images by learning the residuals of a reference image. This global residual learning strategy is beneficial for network training convergence. To learn effective residual features, the network adopts an encoder-feature fusion network-decoder structure. The feature fusion network consists of multiple GDRDBs, used to extract effective local and global features from multiple frames of images, gradually recovering image information and suppressing ghosting. The discriminator is also a neural network used to distinguish whether the image input to the network is a real HDR image or an HDR image generated by the generator.
[0062] Step 2. Preparation of training data
[0063] Three LDR images with different exposure times and their gamma-transformed images are merged into a 6-channel vector. In order to provide more training data and improve training efficiency, all images are randomly cropped into 256×256 image blocks during training as training data, and the pixel values are adjusted to the range of [0, 1].
[0064] Step 3. Pre-train the generator
[0065] The training data is input into the generator, and the content loss function L is selected as the loss function. con :
[0066] L con =|T(Hgt )-T(H pre )|1
[0067] Where T is the tone mapping function, H gt For true HDR images, H pre For the HDR image generated by the generator, ||1 indicates the L1 norm;
[0068] The tone mapping function T is expressed as:
[0069]
[0070] Where H is the image independent variable and μ is the compression coefficient.
[0071] Step 4. Perform adversarial training on the generator and the decision-maker.
[0072] The training data is input into the generator, and the HDR image generated by the generator and the training data are respectively input into the decision unit;
[0073] The generator's adversarial loss function L LSGAN (G) is:
[0074] L LSGAN (G)=E(x)[(D(G(x))-1) 2 |
[0075] Where x is the training data, E represents the expected value, G(x) represents the HDR image generated by the generator from the training data x, and D(G(x)) represents the output result of the HDR image generated by the generator being input into the discriminator;
[0076] The loss function for training the generator is ultimately expressed as L(G):
[0077] L(G)=L con +αL LSGAN (G)
[0078] Here, α is a weight hyperparameter used to balance generative adversarial loss and content loss.
[0079] The loss function for training the discriminator is:
[0080] L(D) = 0.5 × (E(x)[D(G(x))) 2 ]+E(y)[(D(y)-1) 2 ])
[0081] Where y is the real HDR image corresponding to the training data x, and D(y) is the output result of the real HDR image y input to the decision maker.
[0082] Step 5. Input the three real-time LDR images with different exposure times into the trained generator to obtain the final real-time HDR image.
[0083] The training process of the method described in this embodiment is illustrated in the diagram below. Figure 4 As shown, the generator is first pre-trained to possess a certain HDR image generation capability. Then, least-squares adversarial training is used to make the HDR images generated by the generator more consistent with real HDR images in terms of data distribution. This makes the generated HDR images closer to real HDR images in terms of pixel intensity and data distribution. In the pre-training stage, the generator fuses features from three LDR images to generate HDR images. The generated images basically conform to the characteristics of HDR images in terms of content detail and dynamic range. The purpose of this step is to enable the generator, which has a relatively complex structure and a larger number of network parameters, to have a certain HDR image generation capability in advance, avoiding the problems of gradient vanishing or gradient exploding. In the adversarial training stage, the generator already has good generation performance, so the discriminator can more effectively distinguish between real samples and generated samples. The adversarial training between the generator and the discriminator prompts the generator to learn effective update strategies, further improving the generator's performance so that the generated images conform to the data distribution characteristics of real HDR images.
[0084] In the method described in this embodiment, the Generative Adversarial Network (GAN) allows the performance of the generator and discriminator to gradually improve through a game-like interaction between the two networks. During training, the adversarial losses of both the generator and discriminator are simultaneously optimized. The data generated by the generator gradually approaches the real values. When the discriminator can no longer distinguish between real data and data generated by the generator, the adversarial training can be considered to have converged. Traditional GAN networks use the Sigmoid function for activation in the discriminator, and the objective function is the cross-entropy loss function, which is prone to the gradient vanishing problem during training. In this embodiment, LSGAN changes the objective function of GAN from the original cross-entropy loss function to the least squares loss function, making the LSGAN discriminator more sensitive to the differences between real samples and generated samples, thus avoiding the gradient vanishing problem of the discriminator.
[0085] The method described in this embodiment was compared with existing dynamic scene HDR imaging methods on three publicly available datasets. The dataset provided by the existing technology "Deep high dynamic range imaging of dynamic scenes" (referred to as the Kalantari dataset in this embodiment) contains 15 test samples, all of which are foreground motion samples of people against complex backgrounds. This dataset provides real HDR images, and the performance of this method on objective evaluation metrics was compared on this dataset. The datasets provided by the existing technology "An objective deghosting quality metric for HDR images" (referred to as the Tursun dataset in this embodiment) and the datasets provided by the existing technology "Robust patch-based hdrreconstruction of dynamic scenes" (referred to as the Sen dataset in this embodiment) contain samples of scenes with foreground motion of people, background motion, and camera shake. These two datasets do not provide real HDR images, and a subjective visual comparison was performed on these two datasets.
[0086] The method described in this embodiment uses PSNR, SSIM, and HDR-VDP as objective evaluation metrics. PSNR is calculated based on the difference in pixel intensity between images, reflecting the similarity in pixel intensity between the algorithm-generated image and the real image. The SSIM metric comprehensively considers multiple image attributes, and its calculation formula includes three parts: image brightness, contrast, and structure, allowing for a more comprehensive evaluation of image quality. The HDR-VDP standard provides a quality evaluation metric, VDP-Q, which reflects the overall quality level of the generated image relative to the reference image. A higher VDP-Q score indicates a lower probability that the algorithm-generated image and the reference image are visually similar to each other. HDR-VDP also visualizes the perceptible differences between the generated image and the real HDR image using a probability difference map (PMAP).
[0087] To verify the effectiveness of the method described in this embodiment, it was compared with the methods described in the prior art "Robust patch-based hdr reconstruction of dynamic scenes" (referred to as Sen2012 in this embodiment), the methods described in the prior art "Deep high dynamic range imaging of dynamic scenes" (referred to as Kal2017 in this embodiment), the methods described in the prior art "Deep high dynamic range imaging with large foreground motions" (referred to as DeepHDR in this embodiment), the methods described in the prior art "Attention-guided network forghost-free high dynamic range imaging" (referred to as AHDRNet in this embodiment), the methods described in the prior art "Deep HDR imaging via a non-local network" (referred to as NHDRRNet in this embodiment), and the methods described in the prior art "HDR-GAN: HDR image reconstruction from multi-exposed LDR images with large motions" (referred to as HDRGAN in this embodiment).
[0088] Table 1 shows the performance comparison of the above methods on the Kalantari dataset, comparing the PNSR and SSIM metrics in the tone mapping domain (μ domain), uniform coding domain (PU domain), and HDR domain (L domain), as well as the VDP-Q metric.
[0089] Table 1 Performance Comparison Table
[0090]
[0091] As shown in Table 1, the method described in this invention performs well across different data domains, outperforming most existing methods in all metrics, indicating that the method can produce high-quality HDR images. Furthermore, this method achieves optimal values on all SSIM metrics, demonstrating its superior performance in ghosting removal and image information recovery compared to existing methods. A comparison of the results generated by the method of this invention with those of other multi-exposure HDR imaging algorithms on PMAP is shown below. Figure 5As shown in the figure, in detail, the PMAP obtained by the HDR image generated by this method has fewer red pixels compared with other results. This indicates that this method can recover most of the information of the image even when the input image is severely overexposed and does not produce obvious ghosting, generating an image with less blurring distortion and noise.
[0092] Subjective visual comparisons of the generated images using the method described in this invention and existing methods on the Kalantari dataset, Tursun dataset, and Sen dataset are as follows: Figure 6 , Figure 7 and Figure 8 As shown, this paper also includes a comparison with the existing technology "HDR image reconstruction from a single exposure using deep CNNs" (referred to as HDRCNN in this embodiment) and the existing technology "Expandnet: A deep convolutional neural network for high dynamic range expansion from low dynamic range content" (referred to as expandNet in this embodiment). The comparison shows that the method of this invention can generate high-quality HDR images under different lighting conditions and different motion scenes. The foreground and background information of the image are restored, with no obvious ghosting. Compared with the results of other methods, it is closer to a real HDR image, indicating that the method of this invention has excellent ghosting removal and image detail restoration capabilities.
Claims
1. A dynamic scene HDR imaging method based on LSGAN, characterized in that, Includes the following steps: Step 1. Construction of Generative Adversarial Networks Generative adversarial networks consist of a generator and a decision maker; The generator consists of an encoder, a feature fusion network, and a decoder, which are cascaded in sequence. The encoder includes three shallow feature extraction network branches, a first convolutional layer, and a second convolutional layer. Three LDR images with different exposure times are input into the three shallow feature extraction network branches respectively to obtain the shallow features of the three LDR images. The shallow features of the three LDR images are merged by channel and then input into the first convolutional layer and the second convolutional layer in sequence to perform preliminary fusion of the shallow features of the three LDR images. The feature fusion network consists of several cascaded global dilated residual dense connection modules. Each global dilated residual dense connection module consists of cascaded dilated residual dense connection modules and a global branch. Each dilated residual dense connection module includes multiple cascaded dilated convolutional layers and a fifth convolutional layer, which further extract and fuse the initially fused image features. The global branch includes a cascaded global average pooling layer and a sixth convolutional layer, which are used to extract the global vector of the output features of the dilated residual dense connection modules and output it after merging with the output features of the dilated residual dense connection modules. The decoder consists of six convolutional layers and one global residual connection layer. After the first three convolutional layers, bilinear upsampling is cascaded to restore the image to its original size. Then, after two cascaded convolutional layers, the residual image is extracted. The LDR image with the middle exposure time is used as the reference image. The residual image and the reference image are input to the global residual connection layer, added pixel by pixel, and then input to the last convolutional layer to obtain the final generated HDR image. The decision maker consists of four convolutional layers and one fully connected layer cascaded in sequence. The kernels of the four convolutional layers are all 3×3. The first three convolutional layers introduce spectral normalization (SN), and the last three convolutional layers use batch normalization (BN). The activation function of the first three convolutional layers is ReLU. Step 2. Preparation of training data Three LDR images with different exposure times and their gamma-transformed images are merged by channel, randomly cropped into image blocks of size 256×256, and used as training data. The pixel values are adjusted to the range of [0, 1]. Step 3. Pre-train the generator The training data is input into the generator, and the content loss function L is selected as the loss function. con : L con =|T(H gt )-T(H pre )|1 Where T is the tone mapping function, H gt For true HDR images, H pre For the HDR image generated by the generator, ||1 represents the L1 norm; The tone mapping function T is expressed as: Where H is the image independent variable and μ is the compression coefficient; Step 4. Perform adversarial training on the generator and the decision-maker. The training data is input into the generator, and the HDR image generated by the generator and the training data are respectively input into the decision unit; The generator's adversarial loss function L LSGAN (G) is: L LSGAN (G)=E(x)[(D(G(x))-1) 2 ] Where x is the training data, E represents the expected value, G(x) represents the HDR image generated by the generator from the training data x, and D(G(x)) represents the output result of the HDR image generated by the generator being input into the discriminator; The loss function for training the generator is ultimately expressed as L(G): L(G)=L con +αL LSGAN (G) Where α is a weight hyperparameter used to balance generative adversarial loss and content loss; The loss function for training the discriminator is: L(D)=0.5×(E(x)[D(G(x)) 2 ]+E(y)[(D(y)-1) 2 ]) Where y is the real HDR image corresponding to the training data x, and D(y) is the output result of the real HDR image y input to the decision maker; Step 5. Input the three real-time LDR images with different exposure times into the trained generator to obtain the final real-time HDR image.
2. The dynamic scene HDR imaging method based on LSGAN according to claim 1, characterized in that, The shallow feature extraction network branch consists of a third convolutional layer, an attention module, and a fourth convolutional layer, which are cascaded in sequence.
3. The dynamic scene HDR imaging method based on LSGAN according to claim 1, characterized in that, In the encoder, the convolution kernels of the first, second, third, and fourth convolutional layers are 3×3.
4. The dynamic scene HDR imaging method based on LSGAN according to claim 1, characterized in that, In the feature fusion network, the kernels of the dilated convolutional layers are 3×3 with an inflation factor of 2; the kernels of the fifth and sixth convolutional layers are 1×1; each dilated convolutional layer is activated by the Leaky ReLU function.
5. The dynamic scene HDR imaging method based on LSGAN according to claim 1, characterized in that, The kernels of the six convolutional layers in the decoder and the four convolutional layers in the decision layer are both 3×3.
Citation Information
Patent Citations
High dynamic range imaging and ghosting removal method based on generative adversarial network
CN115018733A
Image enhancement method based on improved multi-scale fusion generative adversarial network
CN115223004A