A method, system, device and medium for HDR image reconstruction

Abstract

The application discloses a kind of HDR image reconstruction method, system, equipment and medium, it is related to computer vision image enhancement field, the method includes: obtaining the LDR image of different exposure degree under target scene;Each LDR image is mapped to HDR domain using gamma correction, obtain the gamma correction image of corresponding exposure degree;LDR image and gamma correction image set under target scene are input into image reconstruction model, obtain the HDR image of target scene;Image reconstruction model is obtained using training data to the deep learning model training;Deep learning model includes: feature extraction module and HDR reconstruction module;Feature extraction module is used to extract the double-channel feature containing channel information and spatial information to the input image;HDR reconstruction module is used to reconstruct the input image based on three-pass residual block according to double-channel feature, obtain corresponding HDR image.The application can realize end-to-end ghost-free high-quality HDR image reconstruction.

Description

Technical Field

[0001] This invention relates to the field of computer vision image enhancement, and in particular to an HDR image reconstruction method, system, device and medium. Background Technology

[0002] High Dynamic Range Imaging (HDRI or HDR) is a set of techniques used in computer graphics and cinematography to achieve a greater dynamic range of exposure (i.e., greater difference between light and dark areas) than ordinary digital images. It aims to accurately represent the various brightness variations in a real scene, from areas directly in sunlight to areas in shadow. With the increasing pursuit of high-quality visual experiences, HDR images are widely used in many fields such as film and television, meteorology, gaming, medicine, military, and remote sensing. Therefore, researching HDR image reconstruction methods has significant practical value.

[0003] High dynamic range (HDR) image acquisition technology mainly falls into two categories. One is at the hardware level, primarily involving modifying a single image sensor or using multiple sensors to form an imaging system. However, this method is costly and not widely adopted in practical applications. The other method is at the software level, reconstructing HDR images based on single-exposure or multi-exposure LDR images. However, a single frame contains limited information, and when underexposed or overexposed areas exist in the LDR image, the information in these areas is difficult to repair. Therefore, high-quality HDR images are typically reconstructed based on multi-exposure LDR images.

[0004] Currently, multi-exposure dynamic scene HDR reconstruction algorithms are mainly divided into two categories: traditional methods and deep learning methods. Traditional methods typically remove ghosting in HDR images based on image registration, motion suppression, and patch matching. Image registration-based methods usually search for the most suitable matching region to remove ghosting before fusing the input exposure images. However, this method is prone to misestimation of optical flow under large-scale motion, resulting in ghosting. Motion suppression-based methods typically use an input image as a reference image, first registering the pixels in the LDR image, replacing the misaligned areas with pixels from the reference image or surrounding static pixels, and then fusing the aligned LDR images to generate the HDR image. Although this method can detect some motion region features, it only considers color difference features, leading to a loss of detail. Patch matching-based methods typically first segment the input image into multiple patches, measure the structural similarity of each patch during fusion, and finally iteratively fuse the best-matching patches to obtain the most ideal result. For example, the patch-based iterative optimization matching fusion algorithm has achieved good results, but the complexity of iteration leads to long processing time, and it is prone to failure in scenes with large-scale motion and saturated regions.

[0005] Common methods for acquiring HDR images using deep learning can be divided into CNN-based and Transformer-CNN-based methods. Representative CNN-based methods include: The 2018 DeepHDR algorithm first uses a simple homography transformation to achieve background alignment, then uses a U-Net-based network to generate HDR images from the aligned multi-exposure LDR images. The 2019 AHDRNet algorithm proposes an attention module and a densely connected Dilated Residual Dense Block (DRDB) module for multi-frame alignment and fusion. The 2021 ADNet algorithm uses a spatial attention module to handle the multisaturation problem and a Pyramid Cascading and Deformable (PCD) alignment module to handle misalignment.

[0006] The deep learning-based methods mentioned above have achieved certain results. However, these CNN-based methods, due to their inherent properties such as parameter sharing and small receptive fields, lack global information and cannot effectively solve the ghosting problem in HDR imaging when there is large-scale object motion or undersaturated or oversaturated areas in the LDR image. To address this, the HDR-Transformer algorithm in 2022 combined the advantages of CNN in extracting local information and Transformer in capturing global information, achieving advanced results. However, this method only focuses on space during shallow feature extraction, neglecting channel considerations, which is detrimental to recovering undersaturated or oversaturated areas. Furthermore, the small receptive field of CNN-based branches makes it unsuitable for handling local texture changes caused by small movements.

[0007] In conclusion, how to achieve high-quality HDR image reconstruction remains an urgent problem to be solved. Summary of the Invention

[0008] Based on this, embodiments of the present invention provide an HDR image reconstruction method, system, device, and medium to achieve end-to-end ghost-free, high-quality HDR image reconstruction.

[0009] To achieve the above objectives, embodiments of the present invention provide the following solutions:

[0010] An HDR image reconstruction method, comprising:

[0011] Acquire an LDR image set of the target scene; the LDR image set includes: underexposed LDR images, medium-exposed LDR images, and overexposed LDR images;

[0012] Each image in the LDR image set of the target scene is mapped to the HDR domain using gamma correction to obtain a gamma-corrected image set of the target scene; the gamma-corrected image set includes: underexposed gamma-corrected image, medium-exposed gamma-corrected image and overexposed gamma-corrected image.

[0013] The LDR image set and the gamma-corrected image set of the target scene are input into the image reconstruction model to obtain the HDR image of the target scene;

[0014] The image reconstruction model is obtained by training a deep learning model using training data. The deep learning model includes a feature extraction module and an HDR reconstruction module. The feature extraction module is used to extract dual-channel features containing channel information and spatial information from the input image. The HDR reconstruction module is used to reconstruct the input image based on the dual-channel features and a three-way residual block to obtain the corresponding HDR image.

[0015] Optionally, the feature extraction module includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a first dual attention module, a second dual attention module, and a fourth convolutional layer;

[0016] The first convolutional layer is used to convolve the first merged image to obtain the first shallow feature; the first merged image is the image obtained by merging the input underexposed LDR image and the underexposed gamma-corrected image.

[0017] The second convolutional layer is used to convolve the second merged image to obtain the second shallow layer features; the second merged image is the image obtained by merging the input medium-exposure LDR image and the medium-exposure gamma-corrected image;

[0018] The third convolutional layer is used to convolve the third merged image to obtain the third shallow feature; the third merged image is the image obtained by merging the input overexposed LDR image and the overexposed gamma-corrected image; the first shallow feature and the third shallow feature are non-reference features; the second shallow feature is a reference feature;

[0019] The first dual attention module is used to generate a first dual attention map based on the first shallow features and the second shallow features, using channel attention mechanism and spatial attention mechanism;

[0020] The second dual attention module is used to generate a second dual attention map based on the second shallow features and the third shallow features, using channel attention mechanism and spatial attention mechanism;

[0021] The fourth convolutional layer is used to concatenate the first product feature, the second product feature, and the second shallow feature and then perform a convolution operation to obtain dual-channel features; the first product feature is obtained by multiplying the first dual attention map and the first shallow feature; the second product feature is obtained by multiplying the second dual attention map and the third shallow feature.

[0022] Optionally, the first dual attention module includes: a channel attention module and a spatial attention module;

[0023] The channel attention module is used to sequentially process the first shallow features through global average pooling, two convolution operations, and a sigmoid function activation to obtain a channel attention map;

[0024] The spatial attention module is used to concatenate the first shallow features and the second shallow features, and then perform two convolution operations and a sigmoid function activation on the concatenated features to obtain a spatial attention map.

[0025] The first dual attention map is obtained by multiplying the channel attention map and the spatial attention map.

[0026] Optionally, the HDR reconstruction module includes: a Token Embeddings layer, a first three-way residual block, a second three-way residual block, a third three-way residual block, an AfterConv layer, a global residual layer, and a convolutional module;

[0027] The Token Embeddings layer is used to perform word vector transformation on the dual-channel features to obtain the transformed dual-channel features;

[0028] The first three-way residual block is used to fuse global and local information in the converted dual-channel features to obtain the first fused feature;

[0029] The second three-way residual block is used to fuse the global information and local information in the first fusion feature to obtain the second fusion feature;

[0030] The third three-way residual block is used to fuse the global information and local information in the second fusion feature to obtain the third fusion feature;

[0031] The After Conv layer is used to perform a convolution operation on the third fused feature to obtain the residual output feature;

[0032] The global residual layer is used to perform global residual connection on the residual output features, the dual-channel features and the second shallow layer features to obtain a stitched image;

[0033] The convolution module is used to perform convolution operations and activate the sigmoid function on the stitched image to obtain an HDR image.

[0034] Optionally, the first tee residual block includes: six identical tee residual layers and a Dilated Conv layer connected in sequence; the output of the Dilated Conv layer is connected to the input residual of the first tee residual layer.

[0035] Optionally, the three-way residual layer includes: a global branch based on Transformer and a local branch based on CNN;

[0036] The global branch is used to extract global information from the input features to obtain global features; the local branch is used to extract local information from the input features to obtain local features; the global features and the local features are added together to obtain fused features.

[0037] Optionally, the local branch includes: a first normalization layer and a local information extractor connected in sequence;

[0038] The first normalization layer is used to normalize the input features to obtain normalized features;

[0039] The local information extractor uses a convolution block to perform a convolution operation on the normalized features to obtain a first output feature. Then, it uses three filters of different sizes to extract features from the first output feature to obtain a second, third, and fourth output feature. The second, third, and fourth output features are then concatenated together to obtain a fifth output feature. After a 3×3 convolution, a sixth output feature is obtained. The sixth output feature and the first output feature are then residually concatenated to obtain a seventh output feature. Finally, the seventh output feature is multiplied by the first output feature to obtain the local feature.

[0040] The present invention also provides an HDR image reconstruction system, comprising:

[0041] An image acquisition unit is used to acquire an LDR image set of a target scene; the LDR image set includes: underexposed LDR images, medium-exposed LDR images, and overexposed LDR images;

[0042] The gamma correction unit is used to map each image in the LDR image set of the target scene to the HDR domain using gamma correction, thereby obtaining a gamma-corrected image set of the target scene; the gamma-corrected image set includes: underexposed gamma-corrected image, medium-exposed gamma-corrected image and overexposed gamma-corrected image.

[0043] The reconstruction unit is used to input the LDR image set and the gamma-corrected image set of the target scene into the image reconstruction model to obtain the HDR image of the target scene;

[0044] The image reconstruction model is obtained by training a deep learning model using training data. The deep learning model includes a feature extraction module and an HDR reconstruction module. The feature extraction module is used to extract dual-channel features containing channel information and spatial information from the input image. The HDR reconstruction module is used to reconstruct the input image based on the dual-channel features and a three-way residual block to obtain the corresponding HDR image.

[0045] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the above-described HDR image reconstruction method.

[0046] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described HDR image reconstruction method.

[0047] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0048] This invention provides an image reconstruction model based on a deep learning model. The deep learning model includes a feature extraction module and an HDR reconstruction module. The feature extraction module extracts dual-channel features containing channel information and spatial information from the input image. The HDR reconstruction module reconstructs the input image based on the dual-channel features and three-way residual blocks to obtain the corresponding HDR image. This image reconstruction model has powerful feature extraction capabilities and can achieve end-to-end ghost-free high-quality HDR image reconstruction. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart of an HDR image reconstruction method provided in an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the structure of a deep learning model provided in an embodiment of the present invention;

[0052] Figure 3This is a schematic diagram of the feature extraction module provided in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the structure of the residual layer of the three-way valve provided in an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of an underexposed LDR image with an exposure of -2EV in a publicly available dataset.

[0055] Figure 6 A schematic diagram of a medium-exposure LDR image with an exposure of 0 EV in a publicly available dataset;

[0056] Figure 7 This is a schematic diagram of an overexposed LDR image with an exposure of +2EV in a publicly available dataset.

[0057] Figure 8 This is a schematic diagram of a tone-mapped HDR image of Ground Truth from a publicly available dataset.

[0058] Figure 9 This is a schematic diagram of an HDR image processed by tone mapping using the method of the present invention;

[0059] Figure 10 This is a structural diagram of an HDR image reconstruction system provided in an embodiment of the present invention. Detailed Implementation

[0060] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0061] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Example 1

[0063] The HDR image reconstruction method provided in this embodiment is applicable to the reconstruction of high dynamic range images based on multi-exposure low dynamic range images in dynamic scene applications. See also Figure 1 The HDR image reconstruction method of this embodiment includes:

[0064] Step 101: Obtain the LDR image set of the target scene; the LDR image set includes: underexposed LDR images, medium-exposed LDR images and overexposed LDR images.

[0065] LDR images are low dynamic range imaging (LDR).

[0066] Step 102: Map each image in the LDR image set of the target scene to the HDR domain using gamma correction to obtain the gamma-corrected image set of the target scene.

[0067] The gamma-corrected image set includes: underexposed gamma-corrected images, medium-exposed gamma-corrected images, and overexposed gamma-corrected images.

[0068] Step 103: Input the LDR image set and the gamma-corrected image set of the target scene into the image reconstruction model to obtain the HDR image of the target scene; wherein, the image reconstruction model is obtained by training a deep learning model using training data; the deep learning model includes: a feature extraction module and an HDR reconstruction module.

[0069] The feature extraction module is used to extract dual-channel features containing channel information and spatial information from the input image; the HDR reconstruction module is used to reconstruct the input image based on the dual-channel features and a Triple-Pass Residual Block (TPRB) to obtain the corresponding HDR image.

[0070] The following is combined Figure 2 , Figure 3 and Figure 4 The focus is on introducing the feature extraction module and the HDR reconstruction module.

[0071] See Figure 2 The feature extraction module includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a first dual attention module, a second dual attention module, and a fourth convolutional layer. Figure 2 The three convolutional layers (Conv) in the feature extraction module from top to bottom correspond to the first, second, and third convolutional layers, respectively, while the rightmost convolutional layer (Conv) is the fourth convolutional layer.

[0072] The first convolutional layer is used to convolve the first merged image to obtain the first shallow feature f1; the first merged image is the image obtained by merging the input underexposed LDR image and the underexposed gamma-corrected image.

[0073] The second convolutional layer is used to convolve the second merged image to obtain the second shallow feature f2; the second merged image is the image obtained by merging the input medium-exposure LDR image and the medium-exposure gamma-corrected image.

[0074] The third convolutional layer is used to convolve the third merged image to obtain the third shallow feature f3; the third merged image is the image obtained by merging the input overexposed LDR image and the overexposed gamma-corrected image; the first shallow feature f1 and the third shallow feature f3 are non-reference features; the second shallow feature f2 is a reference feature.

[0075] The first dual attention module is used to generate a first dual attention map m1 based on the first shallow feature f1 and the second shallow feature f2, using channel attention mechanism and spatial attention mechanism.

[0076] The second dual attention module is used to generate a second dual attention map m3 based on the second shallow feature f2 and the third shallow feature f3, using channel attention mechanism and spatial attention mechanism.

[0077] The fourth convolutional layer is used to concatenate the first product feature f'1, the second product feature f'3, and the second shallow feature f2 and then perform a convolution operation to obtain a dual-channel feature F; the first product feature f'1 is obtained by multiplying the first dual attention map m1 and the first shallow feature f1; the second product feature f'3 is obtained by multiplying the second dual attention map m3 and the third shallow feature f3.

[0078] See Figure 3 The upper dual-attention module is the first dual-attention module, and the lower dual-attention module is the second dual-attention module. Both dual-attention modules have the same structure. The following explanation uses the first dual-attention module as an example to illustrate the structure of the dual-attention modules.

[0079] See Figure 3 The first dual attention module includes a channel attention module and a spatial attention module.

[0080] The channel attention module is used to process the first shallow feature f1 sequentially through global average pooling (GAP), two convolution operations (each with a kernel size of 1×1), and activation by a sigmoid function to obtain the channel attention map.

[0081] The spatial attention module is used to concatenate the first shallow feature f1 and the second shallow feature f2, and then perform two convolution operations and a sigmoid function activation on the concatenated features to obtain a spatial attention map.

[0082] The channel attention map and the spatial attention map are multiplied to obtain the first dual attention map m1. Similarly, the second dual attention module finally obtains the second dual attention map m3.

[0083] Please see again Figure 2 The HDR reconstruction module includes: a Token Embeddings layer, a first three-way residual block TPRB1, a second three-way residual block TPRB2, a third three-way residual block TPRB3, an After Conv layer, a global residual layer, and a convolution module.

[0084] The Token Embeddings layer is used to perform word vector transformation on the dual-channel feature F to obtain the transformed dual-channel feature E. The first three-way residual block TPRB1 is used to fuse global and local information in the transformed dual-channel feature E to obtain a first fused feature. The second three-way residual block TPRB2 is used to fuse global and local information in the first fused feature to obtain a second fused feature. The third three-way residual block TPRB3 is used to fuse global and local information in the second fused feature to obtain a third fused feature. The After Conv layer is used to perform a convolution operation on the third fused feature to obtain a residual output feature. The global residual layer is used to perform a global residual connection on the residual output feature, the dual-channel feature, and the second shallow feature to obtain a stitched image. The convolution module is used to perform a convolution operation and sigmoid function activation on the stitched image to obtain an HDR image.

[0085] The three tee residual blocks mentioned above have the same structure. The structure of the tee residual block will be explained below using the first tee residual block TPRB1 as an example.

[0086] Please see again Figure 2 The first three-way residual block TPRB1 includes: six identical three-way residual layers (TPR) connected in sequence and a dilated Conv layer; the output of the dilated Conv layer is connected to the input residual of the first three-way residual layer TPRB1.

[0087] See Figure 4 The three-way residual layer includes: a global branch based on Transformer and a local branch based on CNN.

[0088] The global branch is used to extract global information from the input features to obtain global features; the local branch is used to extract local information from the input features to obtain local features; the global features and the local features are added together to obtain fused features.

[0089] The local branch includes a first normalization layer (LN) and a local information extractor connected in sequence. The first normalization layer (LN) is used to normalize the input features to obtain normalized features. The local information extractor performs a convolution operation on the normalized features using a convolution block to obtain a first output feature T0. Then, it uses three filters of different sizes (1×1, 3×3, and 5×5) to extract features from the first output feature T0 to obtain a second output feature T1, a third output feature T2, and a fourth output feature T3. The second output feature T1, the third output feature T2, and the fourth output feature T3 are then concatenated to obtain a fifth output feature T4. After a 3×3 convolution, a sixth output feature T5 is obtained. The sixth output feature T5 and the first output feature T0 are then residually concatenated to obtain a seventh output feature T6. Finally, the seventh output feature T6 is multiplied by the first output feature T0 to obtain the local features.

[0090] The global branch includes: a second normalized layer (LN), a multi-head self-attention (MSA) module, a third normalized layer (LN), a multilayer perceptron (MLP), and multiple residual connections connected in sequence.

[0091] The following section uses the existing public HDR dataset provided by Kalantari et al. as training data as an example to introduce the training and testing process of the above method in practical applications.

[0092] Step 1: Select the existing public HDR dataset provided by Kalantari et al. This dataset contains 74 training samples and 15 test samples. Each sample contains 3 LDR images with different exposures (denoted as underexposed I1, medium exposed I2, and overexposed I3) and 1 corresponding HDR image (denoted as I). GT ) as GroundTruth. Figures 5-7 The image shows an underexposed LDR image with an exposure of -2EV, a medium-exposure LDR image with an exposure of 0EV, and an overexposed LDR image with an exposure of +2EV from a public HDR dataset.

[0093] Because LDR images are helpful in detecting undersaturated or oversaturated regions, while gamma-corrected images are helpful in detecting alignment issues. Therefore, to better utilize the input data, the LDR image {I i The image {H,i=1,2,3} is mapped to the HDR domain through gamma correction to obtain the gamma-corrected image {H}. i ,i=1,2,3}:

[0094]

[0095] Among them, t iIndicate I i The exposure time is γ, which is the gamma correction parameter.

[0096] Afterwards, data augmentation operations such as rotation, flipping, and scaling can be performed to enrich the dataset samples and improve the robustness of the algorithm.

[0097] Step 2: Extract shallow input features using three convolutional layers respectively. Combine I1 and H1, I2 and H2, I3 and H3 obtained in Step 1 as input to obtain three output features f1, f2 and f3. Use f2 as the reference feature and f1 and f3 as non-reference features.

[0098] Step 3: Use the feature extraction module to extract non-reference features {f} i The parameters {i=1,3} are used as inputs to the reference feature f2 to generate an attention map {m} containing channel information and spatial information. i The attention map {i=1,3} is used to extract bright details in underexposed images and dark details in overexposed images, which is beneficial for recovering undersaturated or oversaturated areas. The obtained attention map is then multiplied element-wise with the corresponding non-reference features to obtain the output feature {f'}. i For each i = 1, 3, the output features f'1 and f'3 are concatenated with the reference feature f2 and then convolved to obtain the output feature F, as shown in the following formula:

[0099]

[0100] in, This indicates a dual attention module.

[0101]

[0102] in, This indicates element-wise multiplication.

[0103] F=Conv[Concat(f′1,f2,f′3)]

[0104] Here, Concat represents the concatenation operation, and Conv represents the convolution operation.

[0105] The internal structure of the dual attention module is described below, using non-reference feature f1 and reference feature f2 as inputs to generate attention map m1 as an example (m3 is similar). The dual attention module includes a channel attention module and a spatial attention module to obtain channel and spatial information, which is beneficial for extracting bright details in underexposed images and dark details in overexposed images, thereby restoring undersaturated or oversaturated regions. First, the non-reference feature f1 is subjected to a global average pooling, and then two 1×1 convolutions and a sigmoid activation to obtain the channel attention map. Then, the non-reference feature f1 is concatenated with the reference feature f2, and the result is obtained by performing two convolutions and a sigmoid activation to obtain the spatial attention map. Finally, the obtained channel attention map Spatial attention map Element-wise multiplication is performed to obtain the double attention map. The formula is as follows:

[0106]

[0107] Where GAP represents global flat pooling, Conv1 represents 1×1 convolution, R represents ReLU activation function, and σ represents Sigmoid activation function.

[0108]

[0109] Where Convat represents the concatenation operation, Conv represents the convolution operation, R represents the ReLU activation function, and σ represents the Sigmoid activation function.

[0110]

[0111] in, This indicates element-wise multiplication.

[0112] Step 4: Use the HDR reconstruction module to fuse global and local features. This module consists of three cascaded three-way residual blocks, an After Conv, and a global residual connection. Each TPRB consists of six cascaded TPRs, a DilatedConv, and a local residual connection. The TPR is the core of this module, divided into a Transformer-based global branch and a CNN-based local branch. This helps to better utilize global and local information to obtain high-quality ghost-free HDR images. The overall workflow of this module is as follows:

[0113] First, a global branch based on the Transformer is constructed, consisting of multiple normalization (LN) modules, multi-head self-attention (MSA) modules, a multilayer perceptron (MLP) module, and multiple residual connections to obtain global information. The output features F obtained in step three are then used for token embeddings to obtain... As input to this branch, the entire process of this branch can be expressed by the following formula:

[0114] G = MSA(LN(E)) + E

[0115] F global =MLP(LN(G))+G

[0116] Among them, F globalThis indicates the global features obtained.

[0117] Then there's the local branching based on CNN, which also involves... As input to this branch, the data passes through a normalization layer and a Local Information Extractor (LIE). The LIE first passes through a convolutional block to obtain the output feature T0. Then, small-sized filters (1×1), medium-sized filters (3×3), and large-sized filters (5×5) are used to extract features T1, T2, and T3, respectively. Small filters can extract local details of the features, while large filters can cover a larger area of ​​the receiver layer, thereby expanding the receptive field of this branch. This helps to better utilize global and local information and fuse them to generate high-quality ghost-free HDR images. Then, T1, T2, and T3 are concatenated together to obtain T4, which is then passed through a 3×3 convolution to obtain T5. This T5 is then residually connected to T0 to obtain T6. Finally, T6 is element-wise multiplied with T0 to obtain the final output feature F of this branch. local The formula is as follows:

[0118] F local =LIE(LN(E))

[0119] Among them, F local This represents the acquired local features.

[0120] Specifically:

[0121] T0 = ​​Conv(LN(E))

[0122] T1 = Conv1(T0)

[0123] T2 = Conv3(T0)

[0124] T3 = Conv5(T0)

[0125] T4 = Concat(T1, T2, T3)

[0126] T5 = Conv3(T4)

[0127] T6 = T5 + T0

[0128]

[0129] Where Conv represents a convolutional block, Conv1 represents a 1×1 convolution, Conv3 represents a 3×3 convolution, Conv5 represents a 5×5 convolution, and Concat represents a concatenation operation. This indicates element-wise multiplication.

[0130] Finally, the obtained global features F global and local features Flobal The output fusion feature is obtained by adding elements one by one. This is used as the input for the next TPR, and so on, until the output of the sixth TPR is fused with features. After passing through a Dilated Conv, a local residual connection is performed with E to obtain the output features. The above process completes the first TPRB. The output features of the first TPRB are then used as the input to the second TPRB, and this process is repeated to obtain the output features of the third TPRB. Then, after an After Conv process, feature F1 is obtained. Finally, F1, F, and f2 are connected globally as residuals, and then subjected to a convolution and sigmoid activation to obtain the final reconstructed HDR image I. H Using the generated I H and label I GT The network is trained using L1 loss and perceptual loss to complete the entire training process for HDR reconstruction, thus generating a model for HDR image reconstruction methods.

[0131] Step 5: Set the parameters of the network model during training and continuously optimize them based on the training process until the best model is trained.

[0132] Specifically, experiments were conducted in the deep learning framework PyTorch. The model was trained for 100 epochs with a batch size of 8, using the ADAM optimizer with an initial learning rate of 2e-4, and β1, β2, and βe-8 were set to 0.9, 0.999, and βe-8, respectively. All experiments were implemented using a single NVIDIA 3090 GPU. HDR images of GroundTruth after tone mapping from the public dataset are shown below. Figure 8 As shown.

[0133] Step Six: Evaluate the model performance of the HDR image reconstruction method. Input the test image into the trained network model (i.e., the image reconstruction model) to obtain the HDR image. Use Photomatix to perform tone mapping on the generated HDR image and display the reconstructed image on an 8-bit display screen. Experimental results show that the HDR image reconstruction method of this invention achieves good results and generates high-quality HDR images. Actual test results are shown in [link to test results]. Figure 9 .

[0134] All the above embodiments have the following advantages: the method no longer uses only a CNN network, but instead uses a network combining Transformer-CNN; the proposed dual attention module includes a channel attention module and a spatial attention module to obtain channel information and spatial information, thereby effectively extracting bright details in underexposed images and dark details in overexposed images; the proposed HDR reconstruction module adopts a dual-branch structure, utilizing the global modeling capability of Transformer in the global branch based on Transformer, and using filters of different sizes to extract features in the local branch based on CNN, expanding the receptive field of this branch. This helps to better utilize global and local information and fuse them to generate high-quality ghost-free HDR images. This method has powerful feature extraction capabilities, achieves end-to-end ghost-free high-quality HDR image reconstruction, and has strong practical value.

[0135] All the above embodiments are applicable to various scenarios, especially those with significant differences in brightness, where the effects are more pronounced. For example, in nighttime photography, in scenes including streetlights, where there are both bright and dark areas, taking multiple photos with different exposures and then merging them can better preserve details in both light and dark areas.

[0136] Example 2

[0137] In order to implement the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, an HDR image reconstruction system is provided below.

[0138] See Figure 10 The system includes:

[0139] Image acquisition unit 201 is used to acquire an LDR image set of the target scene; the LDR image set includes: underexposed LDR images, medium-exposed LDR images and overexposed LDR images.

[0140] The gamma correction unit 202 is used to map each image in the LDR image set of the target scene to the HDR domain using gamma correction to obtain a gamma-corrected image set of the target scene; the gamma-corrected image set includes: underexposed gamma-corrected image, medium-exposed gamma-corrected image and overexposed gamma-corrected image.

[0141] The reconstruction unit 203 is used to input the LDR image set of the target scene and the gamma-corrected image set of the target scene into the image reconstruction model to obtain the HDR image of the target scene.

[0142] The image reconstruction model is obtained by training a deep learning model using training data. The deep learning model includes a feature extraction module and an HDR reconstruction module. The feature extraction module is used to extract dual-channel features containing channel information and spatial information from the input image. The HDR reconstruction module is used to reconstruct the input image based on the dual-channel features and a three-way residual block to obtain the corresponding HDR image.

[0143] Example 3

[0144] This embodiment provides an electronic device, including a memory and a processor. The memory is used to store computer programs, and the processor runs the computer programs to enable the electronic device to perform the HDR image reconstruction method of Embodiment 1.

[0145] Alternatively, the aforementioned electronic device may be a server.

[0146] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the HDR image reconstruction method of Embodiment 1.

[0147] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0148] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An HDR image reconstruction method, characterized in that, include: Obtain the LDR image set of the target scene; The LDR image set includes: underexposed LDR images, medium-exposed LDR images, and overexposed LDR images; Each image in the LDR image set of the target scene is mapped to the HDR domain using gamma correction to obtain a gamma-corrected image set of the target scene; the gamma-corrected image set includes: underexposed gamma-corrected image, medium-exposed gamma-corrected image and overexposed gamma-corrected image. The LDR image set and the gamma-corrected image set of the target scene are input into the image reconstruction model to obtain the HDR image of the target scene; The image reconstruction model is obtained by training a deep learning model using training data. The deep learning model includes a feature extraction module and an HDR reconstruction module. The feature extraction module is used to extract dual-channel features containing channel information and spatial information from the input image. The HDR reconstruction module is used to reconstruct the input image based on the dual-channel features and a three-way residual block to obtain the corresponding HDR image. The feature extraction module includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a first dual attention module, a second dual attention module, and a fourth convolutional layer; The first convolutional layer is used to convolve the first merged image to obtain the first shallow feature; the first merged image is the image obtained by merging the input underexposed LDR image and the underexposed gamma-corrected image. The second convolutional layer is used to convolve the second merged image to obtain the second shallow layer features; the second merged image is the image obtained by merging the input medium-exposure LDR image and the medium-exposure gamma-corrected image; The third convolutional layer is used to convolve the third merged image to obtain the third shallow feature; the third merged image is the image obtained by merging the input overexposed LDR image and the overexposed gamma-corrected image; the first shallow feature and the third shallow feature are non-reference features; the second shallow feature is a reference feature; The first dual attention module is used to generate a first dual attention map based on the first shallow features and the second shallow features, using channel attention mechanism and spatial attention mechanism; The second dual attention module is used to generate a second dual attention map based on the second shallow features and the third shallow features, using channel attention mechanism and spatial attention mechanism; The fourth convolutional layer is used to concatenate the first product feature, the second product feature, and the second shallow feature and then perform a convolution operation to obtain dual-channel features; the first product feature is obtained by multiplying the first dual attention map and the first shallow feature; the second product feature is obtained by multiplying the second dual attention map and the third shallow feature; The HDR reconstruction module includes: a Token Embeddings layer, a first three-way residual block, a second three-way residual block, a third three-way residual block, an After Conv layer, a global residual layer, and a convolutional module; The Token Embeddings layer is used to perform word vector transformation on the dual-channel features to obtain the transformed dual-channel features; The first three-way residual block is used to fuse global and local information in the converted dual-channel features to obtain the first fused feature; The second three-way residual block is used to fuse the global information and local information in the first fusion feature to obtain the second fusion feature; The third three-way residual block is used to fuse the global information and local information in the second fusion feature to obtain the third fusion feature; The After Conv layer is used to perform a convolution operation on the third fused feature to obtain the residual output feature; The global residual layer is used to perform global residual connection on the residual output features, the dual-channel features and the second shallow layer features to obtain a stitched image; The convolution module is used to perform convolution operations and activate the sigmoid function on the stitched image to obtain an HDR image; The first tee residual block includes: six identical tee residual layers and a Dilated Conv layer connected in sequence; the output of the Dilated Conv layer is connected to the input residual of the first tee residual layer; The three-way residual layer includes: a global branch based on Transformer and a local branch based on CNN; The global branch is used to extract global information from the input features to obtain global features; the local branch is used to extract local information from the input features to obtain local features; the global features and the local features are added together to obtain fused features.

2. The HDR image reconstruction method according to claim 1, characterized in that, The first dual attention module includes: a channel attention module and a spatial attention module; The channel attention module is used to sequentially process the first shallow features through global average pooling, two convolution operations, and a sigmoid function activation to obtain a channel attention map; The spatial attention module is used to concatenate the first shallow features and the second shallow features, and then perform two convolution operations and a sigmoid function activation on the concatenated features to obtain a spatial attention map. The first dual attention map is obtained by multiplying the channel attention map and the spatial attention map.

3. The HDR image reconstruction method according to claim 1, characterized in that, The local branch includes: a first normalization layer and a local information extractor connected in sequence; The first normalization layer is used to normalize the input features to obtain normalized features; The local information extractor uses a convolution block to perform a convolution operation on the normalized features to obtain a first output feature. Then, it uses three filters of different sizes to extract features from the first output feature to obtain a second, third, and fourth output feature. The second, third, and fourth output features are then concatenated together to obtain a fifth output feature. After a 3×3 convolution, a sixth output feature is obtained. The sixth output feature and the first output feature are then residually concatenated to obtain a seventh output feature. Finally, the seventh output feature is multiplied by the first output feature to obtain the local feature.

4. An HDR image reconstruction system, characterized in that, include: Image acquisition unit, used to acquire LDR image set of target scene; The LDR image set includes: underexposed LDR images, medium-exposed LDR images, and overexposed LDR images; The gamma correction unit is used to map each image in the LDR image set of the target scene to the HDR domain using gamma correction, thereby obtaining a gamma-corrected image set of the target scene; the gamma-corrected image set includes: underexposed gamma-corrected image, medium-exposed gamma-corrected image and overexposed gamma-corrected image. The reconstruction unit is used to input the LDR image set and the gamma-corrected image set of the target scene into the image reconstruction model to obtain the HDR image of the target scene; The image reconstruction model is obtained by training a deep learning model using training data. The deep learning model includes a feature extraction module and an HDR reconstruction module. The feature extraction module is used to extract dual-channel features containing channel information and spatial information from the input image. The HDR reconstruction module is used to reconstruct the input image based on the dual-channel features and a three-way residual block to obtain the corresponding HDR image. The feature extraction module includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a first dual attention module, a second dual attention module, and a fourth convolutional layer; The first convolutional layer is used to convolve the first merged image to obtain the first shallow feature; the first merged image is the image obtained by merging the input underexposed LDR image and the underexposed gamma-corrected image. The second convolutional layer is used to convolve the second merged image to obtain the second shallow layer features; the second merged image is the image obtained by merging the input medium-exposure LDR image and the medium-exposure gamma-corrected image; The third convolutional layer is used to convolve the third merged image to obtain the third shallow feature; the third merged image is the image obtained by merging the input overexposed LDR image and the overexposed gamma-corrected image; the first shallow feature and the third shallow feature are non-reference features; the second shallow feature is a reference feature; The first dual attention module is used to generate a first dual attention map based on the first shallow features and the second shallow features, using channel attention mechanism and spatial attention mechanism; The second dual attention module is used to generate a second dual attention map based on the second shallow features and the third shallow features, using channel attention mechanism and spatial attention mechanism; The fourth convolutional layer is used to concatenate the first product feature, the second product feature, and the second shallow feature and then perform a convolution operation to obtain dual-channel features; the first product feature is obtained by multiplying the first dual attention map and the first shallow feature; the second product feature is obtained by multiplying the second dual attention map and the third shallow feature; The HDR reconstruction module includes: a Token Embeddings layer, a first three-way residual block, a second three-way residual block, a third three-way residual block, an After Conv layer, a global residual layer, and a convolutional module; The Token Embeddings layer is used to perform word vector transformation on the dual-channel features to obtain the transformed dual-channel features; The first three-way residual block is used to fuse global and local information in the converted dual-channel features to obtain the first fused feature; The second three-way residual block is used to fuse the global information and local information in the first fusion feature to obtain the second fusion feature; The third three-way residual block is used to fuse the global information and local information in the second fusion feature to obtain the third fusion feature; The After Conv layer is used to perform a convolution operation on the third fused feature to obtain the residual output feature; The global residual layer is used to perform global residual connection on the residual output features, the dual-channel features and the second shallow layer features to obtain a stitched image; The convolution module is used to perform convolution operations and activate the sigmoid function on the stitched image to obtain an HDR image; The first tee residual block includes: six identical tee residual layers and a Dilated Conv layer connected in sequence; the output of the Dilated Conv layer is connected to the input residual of the first tee residual layer; The three-way residual layer includes: a global branch based on Transformer and a local branch based on CNN; The global branch is used to extract global information from the input features to obtain global features; the local branch is used to extract local information from the input features to obtain local features; the global features and the local features are added together to obtain fused features.

5. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the HDR image reconstruction method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the HDR image reconstruction method as described in any one of claims 1 to 3.

Images