A raw domain-based dual-branch hdr video reconstruction method

Abstract

The application discloses a kind of double-branch HDR video reconstruction methods based on Raw domain, belong to video signal processing technical field;A kind of double-branch HDR video reconstruction methods based on Raw domain, specifically includes the following steps: S1, establishes the Raw domain video HDR dataset of synthesis;S2, designs double-branch Raw video HDR reconstruction algorithm based on S1;S3, training model;S4, the low dynamic range (LDR) of Raw video sequence in test set is input into model, and the corresponding high dynamic range output result is obtained;The first video HDR dataset of simulating real noise distribution is synthesized in Raw domain in the application, and the training and evaluation of HDR reconstruction method under night or extreme scene are provided with benchmark dataset;Meanwhile, the dynamic range expansion of noisy LDR video under difficult scene is realized using the proposed content enhancement module.

Description

Technical Field

[0001] This invention relates to the field of video signal processing technology, and in particular to a dual-branch HDR video reconstruction method based on the Raw domain. Background Technology

[0002] High Dynamic Range (HDR) technology utilizes multiple Low Dynamic Range (LDR) images taken at different exposures to extend the dynamic range of an image, enriching image details and improving image contrast. The irradiance of a scene in natural conditions can range from... The dynamic range of a camera is not equal to that of the scene. Ordinary cameras may only record photos with a bit depth of 8 or 10 bits, which cannot fully capture the brightness range of the scene. When the dynamic range captured by the camera is smaller than the dynamic range of the scene, it may cause overexposed or underexposed areas in the captured image, affecting visual quality.

[0003] With the development of HDR technology, the approach has gradually shifted from traditional statistical fusion methods to deep learning-based methods for reconstructing HDR images. These deep learning methods typically involve two steps: first, aligning LDR images from different times and exposures; and second, fusing the aligned LDR images into an HDR image. Compared to image HDR reconstruction, video HDR reconstruction requires reconstructing the HDR result of each frame in the original LDR frame sequence. Existing methods often target alternating exposure video sequences (e.g., -2EV, +2EV, -2EV, ...), employing a sliding window approach to input three or five adjacent LDR frames with different exposures, aligning and fusing them to reconstruct the HDR result of the intermediate frames.

[0004] Previous video HDR reconstruction methods often directly processed sRGB images. However, sRGB images have undergone complex image processing pipelines (ISPs) within the camera, such as black level correction, depigmentation, white balance, gamma correction, and color gamut conversion. This not only causes sRGB images to lose some original information, but also makes HDR reconstruction more difficult due to some non-linear mapping operations. Utilizing the raw domain data output from the camera sensor can effectively solve these problems. Raw domain data contains a wider bit depth, includes richer scene information, and, because it is not affected by subsequent ISP processing, has better linearity, making it more conducive to HDR reconstruction.

[0005] On the other hand, for HDR reconstruction of extremely dark scenes, it is often necessary to consider the severe noise effects. Raw domain data can more accurately model noise, thus enabling the model to better learn denoising of real scenes, and is widely used in image and video denoising tasks. At the same time, existing methods, such as Deep HDR video from sequences with alternating exposures proposed by Kalantari et al. (Computer Graphics Forum 38, 193–205 (2019)), lack special design for noise in extremely dark areas, resulting in poor HDR performance of the network in reconstructing noisy images. HDR video reconstruction: A coarse-to-fine network and a real-world benchmark dataset proposed by Chen et al. (In: Proceedings of the IEEE / CVF International Conference on ComputerVision. pp. 2502–2511 (2021)) uses a two-stage processing alignment and fusion, which over-smooths details when using short-exposure images as references, while retaining noise when using long-exposure images as references.

[0006] As can be seen from the above, deep learning-based HDR image and video reconstruction methods are often limited by noise. Existing methods mainly train images in the sRGB domain, which cannot model real noise conditions. At the same time, the network structure is mainly designed for alignment and fusion, lacking a module for processing noise, resulting in low HDR reconstruction quality in difficult scenarios. In order to solve the above problems, this invention proposes a dual-branch HDR video reconstruction method based on the Raw domain. Summary of the Invention

[0007] The purpose of this invention is to establish a Raw video HDR dataset and, based on this dataset, propose a Raw domain-based dual-branch HDR video reconstruction method to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A method for reconstructing dual-branch HDR video based on the Raw domain specifically includes the following steps:

[0010] S1. Create a synthetic Raw domain video HDR dataset: Create a Raw dataset, specifically including the following:

[0011] S101. Source sRGB video data selection: 21 sRGB domain HDR videos and the high-quality video dataset Vimeo-90K, captured by Froehlich and Kronander et al., were selected as source sRGB video data. Each HDR video was re-exposed using selected exposure parameters to simulate an alternating exposure LDR video sequence;

[0012] S102, Raw Dataset Establishment: By simulating the camera imaging pipeline, sRGB video is converted into a Raw video dataset. Specific steps include: inverting the camera response (CRF) curve, simulating Bayer format, data augmentation, re-exposure, and adding noise to obtain HDR and LDR images in the Raw domain, which are then used as training data pairs in the Raw domain.

[0013] S2. Design Reconstruction Algorithm: Based on the data pairs obtained in S1, use LDR Raw frames. and HDR Raw Frames As training pairs, a dual-branch Raw video HDR reconstruction algorithm was designed.

[0014] S3. Model Training: The model is built based on the reconstruction algorithm designed in S2, and trained using the deep learning framework PyTorch platform. The model is iterated for 15 epcohs on the entire dataset, and then the learning rate is reduced to 0.00001. The iteration continues until the loss converges to obtain the final model.

[0015] S4. Output Results: Input the low dynamic range Raw video sequences from the test set into the final model obtained in S3 to obtain the corresponding high dynamic range output results.

[0016] Preferably, step S102, which involves simulating a camera imaging process to convert sRGB video to Raw video, specifically includes the following steps:

[0017] S1021. For the Vimeo-90K dataset, the video frames are transformed from the non-linear domain to the linear domain by estimating the CRF curve.

[0018] S1022. Downsample the 3-channel sRGB frame to four channels at a quarter resolution: red, green, green, and blue, and combine them into a mosaic image according to the GRBG Bayer format.

[0019] S1023. Convert the mosaic image into a 4-channel image of G, R, B, G, and then randomly scale, translate, and rotate it;

[0020] S1024. Convert the Raw domain HDR frame to the Raw domain LDR frame according to specific exposure parameters.

[0021] S1025. Add simulated Gaussian and Poisson noise to the Raw domain LDR frame.

[0022] Preferably, the design of the dual-branch Raw video HDR reconstruction algorithm mentioned in S2 specifically includes the following steps:

[0023] S201, Noise estimation: Each input consists of three consecutive raw domain LDR frames. , , Noise level map is estimated using a noise estimation network:

[0024]

[0025] S202, Data Processing and Feature Extraction: Input three consecutive raw domain LDR frames , , and its corresponding exposure factor , , Exposure correction is performed on the input LDR image using the exposure coefficient. The correction formula is as follows:

[0026]

[0027] Use the above formula to input Mapped to the same exposure level;

[0028] Then, the feature extraction module uses convolution to extract features:

[0029]

[0030] in, Indicates the first i The frame-extracted features, the input LDR image is used to help detect overexposed and underexposed areas, the input HDR image is used to help with subsequent alignment, and the noise level image helps detect noise areas;

[0031] S203, Feature Alignment: A cascaded pyramid-shaped deformable convolutional structure first downsamples the input features twice to obtain features at multiple scales.

[0032]

[0033] in, Indicates downsampling;

[0034] At the s-th scale, using Features of intermediate frames Cascaded Estimation Offset :

[0035]

[0036] The calculated offset As the offset of the deformable convolution, the alignment result at the current scale is obtained after processing the features of the previous frame using deformable convolution:

[0037]

[0038]

[0039] in, This indicates a 2x bilinear interpolation upsampling, where the alignment results at each scale are further fused with the alignment results at the previous scale through convolution; the joint prediction from coarse to fine over the feature domain can more accurately estimate displacement at large scales.

[0040] S204, Temporal Fusion: The spatial attention structure obtains the attention correlation between adjacent frames through convolution, helping the network reconstruct ghost-free and accurately exposed HDR images.

[0041]

[0042]

[0043] in, This represents the predicted spatial attention. This indicates element-wise multiplication. Indicates the features after time fusion;

[0044] S205, Content Enhancement Branch: This involves aligning the features... The residual estimation branch (REB) extracts high-frequency information from the input features, helping to recover the missing content from the previous branch.

[0045]

[0046] in, Represents the estimated residual information; REB is the residual estimation branch, which contains multiple dense residual blocks.

[0047] S206, Reconstructing HDR: After a series of residual blocks and jump connections, and The results are then summed and passed through a sigmoid layer to obtain the final raw domain HDR result. ;

[0048] S207, Loss Function: Output after tone mapping truth value Differentiated networks between them:

[0049]

[0050]

[0051] in, For 5000, and This represents the raw domain ground truth image and prediction results after tone mapping.

[0052] Compared with existing technologies, this invention provides a dual-branch HDR video reconstruction method based on the Raw domain, which has the following advantages:

[0053] (1) This invention synthesizes the first video HDR dataset that simulates the real noise distribution in the Raw domain, providing a benchmark dataset for the training and evaluation of HDR reconstruction methods in night or extreme scenes;

[0054] (2) Based on the proposed Raw video HDR dataset, this invention proposes a dual-branch HDR video reconstruction method. At the same time, by utilizing the proposed deformable convolution alignment module and content enhancement module, the dynamic range of noisy LDR video in difficult scenes is extended.

[0055] (3) A comparative experiment was conducted between the reconstruction algorithm proposed in this invention and the mainstream reconstruction methods on the market. The results show that the reconstruction algorithm proposed in this invention is superior to the current mainstream HDR reconstruction methods based on sRGB, and is superior to or equivalent to the results of directly converting the mainstream methods to the Raw domain. Through the research and exploration of this invention, it is hoped that it can inspire more research on video HDR reconstruction methods based on the Raw domain. Attached Figure Description

[0056] Figure 1 This is a flowchart of the algorithm for a dual-branch HDR video reconstruction method based on the Raw domain proposed in this invention.

[0057] Figure 2 This is a visual comparison of the results of the proposed dual-branch HDR video reconstruction method based on the Raw domain with other video / image HDR reconstruction algorithms on a test set. Detailed Implementation

[0058] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0059] Example 1:

[0060] A method for reconstructing dual-branch HDR video based on the Raw domain specifically includes the following steps:

[0061] S1. Create a synthetic Raw domain video HDR dataset: Create a Raw dataset, specifically including the following:

[0062] S101. Source sRGB video data selection: 21 sRGB domain HDR videos and the high-quality video dataset Vimeo-90K, captured by Froehlich and Kronander et al., were selected as source sRGB video data. Each HDR video was re-exposed using selected exposure parameters to simulate an alternating exposure LDR video sequence;

[0063] S102, Raw Dataset Establishment: By simulating the camera imaging pipeline, sRGB video is converted into a Raw video dataset. Specific steps include: inverting the camera response (CRF) curve, simulating Bayer format, data augmentation, re-exposure, and adding noise to obtain HDR and LDR images in the Raw domain, which are then used as training data pairs in the Raw domain.

[0064] S102 mentions that the analog camera imaging process converts sRGB video to Raw video, specifically including the following steps:

[0065] S1021. For the Vimeo-90K dataset, the video frames are transformed from the non-linear domain to the linear domain by estimating the CRF curve.

[0066] S1022. Downsample the 3-channel sRGB frame to four channels at a quarter resolution: red, green, green, and blue, and combine them into a mosaic image according to the GRBG Bayer format.

[0067] S1023. Convert the mosaic image into a 4-channel image of G, R, B, G, and then randomly scale, translate, and rotate it;

[0068] S1024. Convert the Raw domain HDR frame to the Raw domain LDR frame according to specific exposure parameters.

[0069] S1025. Add simulated Gaussian and Poisson noise to the Raw domain LDR frame.

[0070] Raw domain data has a different data format than sRGB, so it is necessary to first resample the original RGB three channels to RGGB four channels, and then rearrange them according to the Bayer format to simulate the Raw data format; secondly, Gaussian and Poisson noise are added to the image to simulate the noise distribution in real-world conditions.

[0071] S2. Design Reconstruction Algorithm: Based on the data pairs obtained in S1, use LDR Raw frames. and HDR Raw Frames As training pairs, a dual-branch Raw video HDR reconstruction algorithm was designed.

[0072] S3. Model Training: The model is built based on the reconstruction algorithm designed in S2. The model input consists of 3 frames, and the input video is cropped into 256x256 blocks, with 16 sets of sample data per batch. The Adam optimizer is selected, and the initial learning rate is set to 0.0001. The model is trained using the PyTorch deep learning framework, iterating for 15 epochs on the entire dataset. Then, the learning rate is reduced to 0.00001, and iteration continues until the loss curve converges, yielding the final model.

[0073] S4. Output Results: Input the low dynamic range Raw video sequences from the test set into the final model obtained in S3 to obtain the corresponding high dynamic range output results.

[0074] The first video HDR dataset simulating real noise distribution was synthesized in the Raw domain, providing a benchmark dataset for the training and evaluation of HDR reconstruction methods in nighttime or extreme scenes.

[0075] Example 2:

[0076] Please see Figure 1 Based on Example 1, but with the following differences:

[0077] The design of the dual-branch Raw video HDR reconstruction algorithm mentioned in S2 specifically includes the following steps:

[0078] S201, Noise estimation: Each input consists of three consecutive raw domain LDR frames. , , Noise level map is estimated using a noise estimation network:

[0079]

[0080] S202, Data Processing and Feature Extraction: Input three consecutive raw domain LDR frames , , and its corresponding exposure factor , , Exposure correction is performed on the input LDR image using the exposure coefficient. The correction formula is as follows:

[0081]

[0082] Use the above formula to input Mapped to the same exposure level;

[0083] Then, the feature extraction module uses convolution to extract features:

[0084]

[0085] in, Indicates the first i The frame-extracted features, the input LDR image is used to help detect overexposed and underexposed areas, the input HDR image is used to help with subsequent alignment, and the noise level image helps detect noise areas;

[0086] S203, Feature Alignment: A cascaded pyramid-shaped deformable convolutional structure first downsamples the input features twice to obtain features at multiple scales.

[0087]

[0088] in, Indicates downsampling;

[0089] At the s-th scale, using Features of intermediate frames Cascaded Estimation Offset :

[0090]

[0091] The calculated offset As the offset of the deformable convolution, the alignment result at the current scale is obtained after processing the features of the previous frame using deformable convolution:

[0092]

[0093]

[0094] in, This indicates a 2x bilinear interpolation upsampling, where the alignment results at each scale are further fused with the alignment results at the previous scale through convolution; the joint prediction from coarse to fine over the feature domain can more accurately estimate displacement at large scales.

[0095] S204, Temporal Fusion: The spatial attention structure obtains the attention correlation between adjacent frames through convolution, helping the network reconstruct ghost-free and accurately exposed HDR images.

[0096]

[0097]

[0098] in, This represents the predicted spatial attention. This indicates element-wise multiplication. This represents the features after time fusion.

[0099] S205, Content Enhancement Branch: This involves aligning the features... The residual estimation branch (REB) extracts high-frequency information from the input features, helping to recover the missing content from the previous branch.

[0100]

[0101] in This represents the residual information obtained from the estimation.

[0102] S206, Reconstructing HDR: After a series of residual blocks and jump connections, and The results are then summed and passed through a sigmoid layer to obtain the final raw domain HDR result. ;

[0103] S207, Loss Function: Output after tone mapping truth value Differentiated networks between them:

[0104]

[0105]

[0106] in, For 5000, and This represents the raw domain ground truth image and prediction results after tone mapping.

[0107] Based on the Raw video HDR dataset, this invention proposes a dual-branch HDR reconstruction method. By utilizing the proposed deformable convolution alignment module and content enhancement module, the dynamic range of noisy LDR videos in challenging scenes is extended.

[0108] Example 3:

[0109] Please see Figure 2 Based on Examples 1-2, but with some differences:

[0110] The proposed dual-branch HDR video reconstruction method based on the Raw domain is compared with mainstream methods on the market in the Raw domain. The results on the test set are as follows: Figure 2 As shown in Table 1.

[0111] Table 1 Comparison of Indicators

[0112]

[0113] from Figure 2As can be seen from Table 1, the dual-branch HDR video reconstruction method based on the Raw domain proposed in this invention can better reduce the impact of noise and preserve the original image details through joint learning and information complementarity of the two branches. Combined with the actual image effect and the data in the table, it can be clearly seen that the reconstruction algorithm proposed in this invention has achieved better visual effects and data indicators.

[0114] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A Raw-domain based dual-branch HDR video reconstruction method, characterized in that, Specifically comprising the following steps: S1, establishing a synthetic Raw domain video HDR dataset: establishing a Raw dataset, specifically comprising the following: S101, source sRGB video data selection: selecting several existing sRGB domain HDR videos and high-quality video datasets as source sRGB video data, each HDR video being simulated to expose an alternate exposure LDR video sequence through selected exposure parameters; S102, Raw dataset establishment: converting the sRGB video into a Raw video dataset through a simulated camera imaging pipeline process, the specific steps being: obtaining an inverse camera response curve → simulating a bayer format → data enhancement → re-exposure → adding noise to obtain HDR and LDR images in the Raw domain, the obtained HDR and LDR images in the Raw domain being used as a Raw domain training data pair; S2, design reconstruction algorithm: based on the data pairs obtained in S1, use LDR Raw frames and HDR Raw frames design a dual-branch Raw video HDR reconstruction algorithm as a training pair; S3, model training: building a model based on the reconstruction algorithm designed in S2, and training the model using the deep learning framework Pytorch platform, iterating over the entire dataset for 15 epochs, then reducing the learning rate to 0.00001, continuing to iterate until the loss converges, obtaining the final model; S4, output result: inputting the low dynamic range Raw video sequence in the test set into the final model obtained in S3 to obtain the corresponding high dynamic range output result.

2. The Raw-domain based dual-branch HDR video reconstruction method of claim 1, wherein, The S102 mentioned converts the sRGB video into a Raw video dataset through a simulated camera imaging pipeline, specifically comprising the following steps: S1021, for high-quality video datasets, convert the video frames from a nonlinear domain to a linear domain by estimating the CRF curve; S1022, downsample the 3-channel sRGB frame to four channels: red, green, green, and blue, in a quarter of the resolution, and combine them into a mosaic image according to the GRBG bayer format; S1023, convert the mosaic image into a 4-channel image of G, R, B, and G, and then randomly scale, translate, and rotate it; S1024, convert the Raw domain HDR frame to a Raw domain LDR frame according to specific exposure parameters; S1025, add simulated Gaussian and Poisson noise to the Raw domain LDR frame.

3. The Raw-domain based dual-branch HDR video reconstruction method of claim 1, wherein, The S2 mentioned designs a double-branch Raw video HDR reconstruction algorithm, specifically comprising the following steps: S201, noise estimation: input three consecutive raw domain LDR frames each time , , , estimate the noise level map using the noise estimation network: S202, data processing and feature extraction: input three consecutive raw domain LDR frames 、 、 and its corresponding exposure coefficient 、 、 , exposure correction of the input LDR image using the exposure coefficient, the correction formula is: Using the above equation, the input is mapped to the same exposure level; Then, through a feature extraction module, the features are extracted using convolution: wherein, indicates the i The features of the frame extraction, the input LDR image is used to help detect overexposed and underexposed regions, the input HDR image is used to help subsequent alignment, and the noise level image helps to detect noise regions; S203, feature alignment: a cascaded pyramid deformable convolution structure, which first obtains multiple scale features by down-sampling the input features twice: wherein denotes down-sampling; At the s-th scale, the features are estimated using and the intermediate frame features are concatenated to estimate the offset : The calculated offset As the offset of deformable convolution, the alignment result at the current scale is obtained after processing the last frame feature with deformable convolution: wherein, The bilinear interpolation up-sampling is 2 times, and the alignment results of each scale are further fused through convolution after the alignment results of the previous scale; and the joint prediction from coarse to fine in the feature field can more accurately estimate the displacement under a large scale. S204, temporal fusion: a spatial attention structure that obtains the attention correlation between adjacent frames through convolution, helping the network to reconstruct a ghost-free and exposure-accurate HDR image: wherein, denotes the predicted spatial attention, denotes element-wise multiplication, denotes the time fused features; S205, content enhancement branch: the aligned features are extracted through a residual estimation branch REB to extract the high-frequency information of the input features, helping to restore the missing content in the previous branch: wherein, represents estimated residual information; represents aligned features; S206, reconstruct HDR: After a series of residual blocks, jump connections and Addition, and then through the sigmoid layer to get the final raw domain HDR result ; S207, loss function: output after tone mapping and the true value between the difference network: wherein, is 5000, and denotes the raw domain ground truth image and the prediction result after tone mapping.

Images