Single-exposure raw video hdr reconstruction method and system

Abstract

The present application relates to a kind of single exposure Raw video HDR reconstruction method and system, belong to computer vision and image processing technical field.The method comprises: obtaining single exposure bayer format Raw video sequence;Channel decomposition parallel alignment module is constructed, and input frame is decomposed into full channel flow, red blue channel flow and double green channel flow three parallel branches, using the physical characteristics that red blue channel is not easy to saturate in highlight area and green channel high signal-to-noise ratio, respectively, different feature extraction and alignment are carried out;Exposure complementary mask guided repair module is constructed, by calculating overexposure mask and dividing timing complementary area, using normal exposure information of adjacent frame in complementary area to repair the overexposure area of current frame;Multi-stream features are fused using cross attention mechanism, and high dynamic range video frame is reconstructed by combining mask guided color loss.The present application effectively solves the problem of highlight overflow and dark noise without alternating exposure, avoids ghosting artifacts, and significantly improves the reconstruction quality of HDR video.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to a method and system for HDR reconstruction of single-exposure Raw video. Background Technology

[0002] High Dynamic Range (HDR) imaging technology can record richer brightness levels than Low Dynamic Range (LDR) images, preserving details from direct sunlight to deep shadows, and is widely used in filmmaking, mobile photography, autonomous driving, and virtual reality. However, limited by sensor sensitivity and bit depth, most consumer cameras often face problems of highlight clipping (information loss) and shadow noise amplification in a single exposure.

[0003] To acquire HDR video, current mainstream research mainly focuses on the "alternating exposure" technique, which involves alternating shots of frames at different exposure levels (such as long exposure and short exposure) over time and then merging them. However, this method has significant drawbacks: first, it requires special shooting settings, resulting in a halving of temporal resolution (frame rate); second, the huge differences in brightness between frames can easily lead to registration difficulties in dynamic scenes, resulting in ghosting artifacts.

[0004] To avoid the problems associated with alternating exposures, single-exposure HDR reconstruction has become a research hotspot. Early single-image HDR reconstruction primarily focused on still images, ignoring the temporal correlation of video, and most HDR methods were based on sRGB data processed by an ISP. sRGB data undergoes nonlinear mapping, white balance, and quantization, irreversibly discarding crucial highlight and shadow information. In contrast, raw sensor data has a linear response and high bit depth (10-16 bits), preserving richer radiometric information. However, currently, there is a lack of effective methods to fully utilize the Bayer channel characteristics of raw data and the temporal redundancy in single-exposure videos to recover details in extreme exposure areas.

[0005] Therefore, there is an urgent need to explore an innovative method that can fully utilize the linearity of raw data and the differences in Bayer channels, and achieve high-quality HDR video reconstruction without alternating exposures. Summary of the Invention

[0006] The purpose of this invention is to at least solve the problems of frame rate drop and ghosting caused by the reliance on alternating exposure in existing HDR video reconstruction methods, as well as the serious loss of sRGB data information. It provides a single-exposure Raw video HDR reconstruction method and system, which utilizes the physical characteristics of the Bayer array through a channel decomposition parallel alignment strategy and combines an exposure complementary mask guidance mechanism to repair overexposed areas using timing information, thereby realizing the reconstruction from a Raw video sequence to a high-quality HDR video under single-exposure conditions.

[0007] To achieve the above objectives, the technical solution of the present invention is: a method for HDR reconstruction of single-exposure Raw video, comprising:

[0008] Step S1: Obtain a single-exposure Bayer format Raw video sequence, selecting three consecutive frames as input, namely the previous frame, the current reference frame, and the next frame;

[0009] Step S2: Construct a channel decomposition parallel alignment module to decompose each input frame into three parallel branches: full channel stream, red-blue channel stream, and dual green channel stream; extract features from the three branches respectively, and use optical flow to align the features of adjacent frames to the current reference frame; calculate the exposure weight mask, perform differential weighted fusion of the red-blue channel stream and the dual green channel stream, and finally obtain the features after the three branches are aligned.

[0010] Step S3: Construct an exposure complementary mask-guided repair module, calculate the overexposure mask of each frame and perform motion compensation; divide the spatial region into complementary region and double overexposure region according to the mask difference between the current reference frame and adjacent frames; use the normal exposure information of adjacent frames in the complementary region to repair the overexposure features of the current reference frame through weighted fusion.

[0011] Step S4: Construct a fusion and reconstruction module. Use the cross-attention mechanism to fuse the features obtained in step S3 and the features after branch alignment extracted in step S2. Reconstruct the high dynamic range HDR raw frame through a multi-scale encoder-decoder network.

[0012] Step S5: Construct a loss function that includes mask-guided color loss, and train and optimize the network model using the pre-built RawHDRV dataset.

[0013] Further, step S1 specifically includes:

[0014] Acquiring a single-exposure raw video sequence The data format is RGBG, in three consecutive frames. As input, These represent the previous frame, the current reference frame, and the next frame, respectively; the input data is normalized in the Raw domain while preserving linear response characteristics.

[0015] Furthermore, step S2 specifically includes:

[0016] Each input frame in RGBG format is split into three parallel streams: (1) X stream: containing all four channels, used to maintain global context and structural consistency; (2) RB stream: containing red and blue channels, used for highlight detail recovery; (3) G stream: containing two green channels, used for shadow denoising and detail preservation.

[0017] The optical flow from adjacent frames to the current frame is calculated using an optical flow network, and the adjacent frame features of each flow are warped to obtain aligned features.

[0018] A cross-frame channel attention module is introduced to process the RB stream and G stream, and pixel-level adaptive weights are used to weight the aligned features, finally obtaining the aligned features of the three branches.

[0019] Furthermore, the pixel-level adaptive weights are based on the overexposure mask. Underexposure mask and normal exposure mask Calculations show that for RB streams, an overexposed region is assigned. Higher weighting is applied to take advantage of the red and blue channels' resistance to saturation in highlight areas; for the G stream, underexposed areas are given higher weighting. Higher weighting is used to take advantage of the high signal-to-noise ratio of the green channel.

[0020] Furthermore, step S3 specifically includes:

[0021] Calculate the overexposure mask for each frame. The system uses optical flow to transform the masks of adjacent frames to the coordinate system of the current reference frame; it then compares the mask of the current reference frame with the transformed masks of adjacent frames to delineate complementary regions. and double overexposure regions; based on complementary regions Calculate the fusion weights, enhance the weights of features from adjacent frames in complementary regions to correct the overexposure features of the current reference frame; reduce the weights of features from the current reference frame and adjacent frames in double overexposure regions to suppress error propagation.

[0022] Furthermore, the complementary region Defined as: the area where the current reference frame is overexposed, but adjacent frames are in the same position and are in the normal exposure state.

[0023] Furthermore, the dual overexposure region is defined as: the region where both the current reference frame and the adjacent frame are in an overexposure state.

[0024] Furthermore, in step S4, in the multi-scale encoder-decoder network, the encoder adopts a three-layer downsampling path, and the decoder adopts a symmetrical upsampling path and introduces cross-layer skip connections.

[0025] Furthermore, in step S5, the loss function includes L1 loss, logarithmic domain L2 loss, and mask-guided color loss; the mask-guided color loss includes: (1) mask normalization L1 term: calculate the L1 distance between the prediction result and the true value only in the overexposed mask area, and normalize it using the mask area to force the network to focus on highlight recovery; (2) gradient smoothing term: calculate the gradient of the prediction result in the horizontal and vertical directions, constrain the spatial continuity of the highlight boundary, and reduce artifacts.

[0026] The present invention also provides a single-exposure Raw video HDR reconstruction system for performing the single-exposure Raw video HDR reconstruction method as described above, comprising:

[0027] The data acquisition module is responsible for acquiring the Raw video sequence captured in single-exposure mode;

[0028] The data processing module is responsible for performing channel decomposition and parallel alignment, extracting and matching features of the full channel stream, red-blue channel stream, and dual green channel stream; utilizing exposure complementarity to repair overexposed areas of the current reference frame using effective information from adjacent frames; fusing multi-stream features and outputting the final HDR video frame.

[0029] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the single-exposure Raw video HDR reconstruction method as described above.

[0030] The present invention also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the single-exposure Raw video HDR reconstruction method as described above.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) This invention proposes a parallel processing architecture based on channel decomposition, which for the first time systematically utilizes the differences in physical characteristics of different color channels in the Bayer array of Raw images in terms of noise and saturation (i.e., red and blue channels are beneficial for highlights and green channels are beneficial for shadows), significantly improving the ability to recover details under extreme exposure conditions.

[0033] (2) The present invention introduces an exposure complementary mask guidance mechanism, which cleverly utilizes the temporal redundancy in the video sequence, that is, it uses the "position-exposure" change brought about by the movement of objects or cameras to borrow unsaturated pixel information from adjacent frames to repair the overexposed area of ​​the current frame, effectively solving the problem of complete loss of highlight information under single exposure.

[0034] (3) The present invention designs a mask-guided color loss function to specifically constrain the chromaticity consistency and gradient smoothness of the highlight region, effectively eliminating the common color cast and artifact problems in highlight reconstruction;

[0035] (4) This invention eliminates the reliance on alternating exposure shooting, avoids frame rate loss and ghosting problems in dynamic scenes, and provides a feasible solution for real-time high-quality HDR video acquisition on mobile devices. Attached Figure Description

[0036] Figure 1 This is a flowchart of a single-exposure Raw video HDR reconstruction method provided by the present invention;

[0037] Figure 2 This is an overall architecture diagram of the RawHDRV network in this invention, showing the connection relationship between channel decomposition parallel alignment and mask-guided repair;

[0038] Figure 3 This is a schematic diagram of the Channel Decomposition Parallel Alignment (CDPA) module in this invention;

[0039] Figure 4 This is a schematic diagram of the principle of the complementary mask guided repair (MGR) module in this invention, illustrating the logic of the division of complementary regions;

[0040] Figure 5 The images show the results of reconstructing HDR images on the RawHDRV dataset using the method of this invention. (a) is a three-frame low dynamic range (LDR) video sequence input to the model, and (b) is the high dynamic range (HDR) image reconstructed after processing by the model, corresponding to the center frame. Detailed Implementation

[0041] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0042] This invention provides a method for HDR reconstruction of single-exposure Raw video, comprising:

[0043] Step S1: Obtain a single-exposure Bayer format Raw video sequence, selecting three consecutive frames as input, namely the previous frame, the current reference frame, and the next frame;

[0044] Step S2: Construct a channel decomposition parallel alignment module to decompose each input frame into three parallel branches: full channel stream, red-blue channel stream, and dual green channel stream; extract features from the three branches respectively, and use optical flow to align the features of adjacent frames to the current reference frame; calculate the exposure weight mask, perform differential weighted fusion of the red-blue channel stream and the dual green channel stream, and finally obtain the features after the three branches are aligned.

[0045] Step S3: Construct an exposure complementary mask-guided repair module, calculate the overexposure mask of each frame and perform motion compensation; divide the spatial region into complementary region and double overexposure region according to the mask difference between the current reference frame and adjacent frames; use the normal exposure information of adjacent frames in the complementary region to repair the overexposure features of the current reference frame through weighted fusion.

[0046] Step S4: Construct a fusion and reconstruction module. Use the cross-attention mechanism to fuse the features obtained in step S3 and the features after branch alignment extracted in step S2. Reconstruct the high dynamic range HDR raw frame through a multi-scale encoder-decoder network.

[0047] Step S5: Construct a loss function that includes mask-guided color loss, and train and optimize the network model using the pre-built RawHDRV dataset.

[0048] The following are specific implementation examples of the present invention.

[0049] Example 1:

[0050] This embodiment discloses a single-exposure Raw video HDR reconstruction method. This method directly processes Raw sensor data and utilizes Bayer channel characteristics and temporal complementarity to recover high dynamic range. See appendix. Figure 1 The above method includes the following steps:

[0051] Step S1: Obtain the target Raw video sequence and preprocess it.

[0052] Acquiring a single-exposure raw video sequence The data format is RGBG (four-channel Bayer mosaic). This invention uses three consecutive frames... As input, the aim is to reconstruct the intermediate frame. Corresponding HDR image The input data is normalized in the Raw domain while preserving linear response characteristics.

[0053] Step S2: Construct a Channel Decomposition Parallel Alignment (CDPA) module to extract and align features.

[0054] like Figure 2 and Figure 3 As shown, considering that green pixels in Bayer sensors have higher photon capture efficiency and signal-to-noise ratio (suitable for dark areas), while red and blue pixels are less prone to saturation in bright areas (suitable for highlights), this embodiment designs a three-way parallel channel (branch) stream:

[0055] (1) X-stream (full channel): Input complete 4-channel data and extract features. It is used to maintain global structure and context information;

[0056] (2) RB stream (red and blue channels): Only input the Red and Blue channels to extract features. Focusing on highlight restoration;

[0057] (3) G (G1G2) stream (dual green channel): Input two green channels and extract features. ( ), focusing on noise reduction in dark areas.

[0058] Calculate the optical flow from adjacent frames to intermediate frames using a pre-trained optical flow network (such as SPyNet). And perform a warping operation on the features of adjacent frames in each stream to obtain aligned features:

[0059]

[0060] To eliminate registration errors caused by optical flow residuals, this invention introduces a cross-frame channel cross-attention module to perform feature fusion between the aligned features of adjacent frames and the features of the center frame.

[0061]

[0062] in This represents the complementary features of attention-weighted CA. This operation is performed independently in the three channel streams to preserve channel-level characteristics and local exposure differences.

[0063] To further utilize channel characteristics, an adaptive weighting based on the exposure mask is introduced. First, the overexposure mask is calculated. (e.g., pixel value > 0.95) and underexposure mask (e.g., pixel value < 0.05), and normal exposure mask. .

[0064] Applying differential weights to the RB and G flows is represented as follows:

[0065]

[0066]

[0067] in Then, the weighted channel features are output:

[0068]

[0069] Where ⊙ represents element-wise multiplication. This design embodies the principle of channel-differential weighting: in overexposed areas, the RB flow is enhanced to recover highlights; in underexposed areas, the G flow is increased to enhance shadow structure; and in normal areas, the contributions of the three flows are balanced.

[0070] Step S3: Construct an exposure complementary mask guided repair module (MGR) to repair overexposed areas.

[0071] like Figure 4 As shown, due to the movement of the camera or the object being photographed, the current (intermediate) frame... Medium-saturated (overexposed) pixels in adjacent frames or The corresponding position may not be saturated. This step utilizes this characteristic to perform repair.

[0072] First, calculate the overexposure mask for each frame. And masking adjacent frames through optical flow field Perform a warping operation to the center frame coordinate system. Taking forward propagation as an example, for frame... overexposure mask :

[0073]

[0074] By comparing the current frame mask With propagation mask The space is divided into three categories:

[0075] in Represents element-wise product. , This is the mask after binarization. For binarization operation, The threshold value is set to 0.5.

[0076] (1) Complementary regions ( ( ): The current frame is overexposed, but adjacent frames are normal. This is the most valuable area for restoration.

[0077] (2) Double overexposure zone ( Both frames are overexposed. This needs to be suppressed to prevent the introduction of artifacts.

[0078] Based on the above division, the fusion weight is calculated. and The process is represented as follows:

[0079]

[0080]

[0081] in (Normal frame enhancement factor) and (Overexposure frame suppression factor). This strategy significantly enhances the contribution of normal frames in complementary regions and significantly suppresses overexposure frame interference, while simultaneously attenuating both in double-overexposure regions to avoid error propagation. The above weighting operation applies to the feature pairs after the warping operation. Output Enhancement Features The data is then fed into a cross-frame attention module for fusion.

[0082]

[0083] The whole process is forward ( ) and backward ( The two directions are executed independently, and two propagation features are finally obtained at the center frame. After being concatenated with multi-channel features, it is then convolved with a 3×3 convolution. Integrated into:

[0084]

[0085] in This indicates channel-level splicing. The data is then input into the reconstruction module to ensure maximum utilization of time context information.

[0086] Step S4: Fusion and Reconstruction.

[0087] Multi-channel and temporal augmentation features are input into a multi-scale encoder-decoder architecture for feature integration and high-level semantic reconstruction. The encoder employs a three-layer downsampling path to capture global contextual information at different scales: from level 1 to level 3, [2, 3, 4] TransformerBlock stacks are configured respectively. The decoder employs a symmetrical upsampling path and introduces cross-layer skip connections to preserve local structural details. Finally, the output features are refined and reconstructed into HDRAW frames through convolutional layers, and then connected to the center frame via residual connections. Fusion:

[0088]

[0089] in This represents the combination of an encoder-decoder and a refining module. This design ensures that the model maintains both luminance gamut expansion and texture detail fidelity during reconstruction.

[0090] Step S5: Model training and loss function calculation.

[0091] This invention uses a combined loss function during the training phase:

[0092]

[0093] The mask-weighted color loss weight λ is set to 0.3. L1 loss. Ensure overall brightness consistency; logarithmic domain L2 loss In the logarithmic domain, the weights of bright and dark areas are balanced.

[0094] The key innovation lies in mask-guided color loss. The calculation formula is as follows:

[0095]

[0096] in, and These represent the HDR frames predicted by the network and their corresponding ground truth HDR annotations, respectively. and Predicting a section of the HDR gradient in the horizontal and vertical directions respectively, used to characterize the local continuity of brightness and color changes. (Symbol) This represents the pixel-by-pixel inner product operation. for Norm, It indicates the expectation of spatial dimensions; , , Ensure the values ​​are stable.

[0097] The first method utilizes an overexposure mask. The normalization error of the highlight region is calculated only, forcing the network to focus on restoring the highlight color; the second term is a gradient smoothing term to prevent discontinuities or artifacts at the highlight boundaries.

[0098] This embodiment also provides a single-exposure Raw video HDR reconstruction system based on the above method, including:

[0099] The data acquisition module is configured to connect to an image sensor and read Bayer format Raw video data streams;

[0100] The data processing module has built-in parallel convolutional neural network branches for processing full-band, long-band (red-blue), and medium-band (green) signals respectively; it includes a logic operation unit for performing mask generation, optical flow transformation, and region logic judgment, and performing weighted operations; and it includes a Transformer calculation unit for performing deep feature fusion and image reconstruction.

[0101] This invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform any of the methods described in the embodiments.

[0102] This invention provides a computing device, including a memory and a processor. The memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the embodiments.

[0103] The memory may include high-speed random access memory (RAM) or non-volatile memory. The processor may be an integrated circuit chip, including a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), or field-programmable gate array (FPGA), etc.

[0104] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A single-exposure Raw video HDR reconstruction method, characterized in that, include: Step S1: Obtain a single-exposure Raw video sequence, selecting three consecutive frames as input, namely the previous frame, the current reference frame, and the next frame; Step S2: Construct a channel decomposition parallel alignment module to decompose each input frame into three parallel branches: full channel stream, red-blue channel stream, and dual green channel stream; Feature extraction is performed on the three branches respectively, and optical flow is used to align the features of adjacent frames to the current reference frame; by calculating the exposure weight mask, the red-blue channel stream and the dual green channel stream are differentially weighted and fused to finally obtain the features after the three branches are aligned. Step S3: Construct an exposure complementary mask-guided repair module, calculate the overexposure mask for each frame and perform motion compensation; based on the mask difference between the current reference frame and adjacent frames, divide the spatial region into complementary regions and double overexposure regions. By utilizing the normal exposure information of adjacent frames in complementary regions, the overexposure characteristics of the current reference frame are repaired through weighted fusion. Specifically, it includes: Calculate the overexposure mask for each frame. The system uses optical flow to transform the masks of adjacent frames to the coordinate system of the current reference frame; it then compares the mask of the current reference frame with the transformed masks of adjacent frames to delineate complementary regions. and double overexposure regions; based on complementary regions Calculate the fusion weights, enhance the weights of features from adjacent frames in complementary regions, and correct the overexposure features of the current reference frame; reduce the weights of features from the current reference frame and adjacent frames in double overexposure regions. The complementary region defined as: a region where the current reference frame is in overexposed state, but the adjacent frame is in normal exposure state at the same location; Step S4: Construct a fusion and reconstruction module. Use the cross-attention mechanism to fuse the features obtained in step S3 and the features after branch alignment extracted in step S2. Reconstruct high dynamic range (HDRRaw) frames through a multi-scale encoder-decoder network. Step S5: Construct a loss function that includes mask-guided color loss, and train and optimize the network model using the pre-built RawHDRV dataset.

2. The single-exposure Raw video HDR reconstruction method of claim 1, wherein, Step S1 specifically includes: Acquiring a single-exposure raw video sequence The data format is RGBG, in three consecutive frames. As input, These represent the previous frame, the current reference frame, and the next frame, respectively; the input data is normalized in the Raw domain while preserving linear response characteristics.

3. The single-exposure Raw video HDR reconstruction method of claim 1, wherein, Step S2 specifically includes: Each input frame in RGBG format is split into three parallel streams: (1) X stream: containing all four channels; (2) RB stream: containing the red and blue channels; (3) G stream: containing two green channels; The optical flow from adjacent frames to the current frame is calculated using an optical flow network, and the adjacent frame features of each flow are warped to obtain aligned features. A cross-frame channel attention module is introduced to process the RB stream and G stream, and pixel-level adaptive weights are used to weight the aligned features, finally obtaining the aligned features of the three branches.

4. The single-exposure Raw video HDR reconstruction method of claim 3, wherein, The pixel-level adaptive weights are based on the overexposure mask. Underexposure mask and normal exposure mask Calculated.

5. The single-exposure Raw video HDR reconstruction method of claim 1, wherein, In step S4, in the multi-scale encoder-decoder network, the encoder adopts a three-layer downsampling path, and the decoder adopts a symmetrical upsampling path and introduces cross-layer skip connections.

6. The single-exposure Raw video HDR reconstruction method of claim 1, wherein, In step S5, the loss function includes L1 loss, logarithmic domain L2 loss and mask-guided color loss; the mask-guided color loss includes: (1) mask normalization L1 term: calculate the L1 distance between the prediction result and the true value only in the overexposed mask area, and normalize it using the mask area; (2) gradient smoothing term: calculate the gradient of the prediction result in the horizontal and vertical directions.

7. A single-exposure Raw video HDR reconstruction system for performing the single-exposure Raw video HDR reconstruction method according to any one of claims 1-6, characterized in that, include: The data acquisition module is responsible for acquiring the Raw video sequence captured in single-exposure mode; The data processing module is responsible for performing channel decomposition and parallel alignment, extracting and analyzing the features of the complete channel stream, red-blue channel stream, and dual-green channel stream. By leveraging exposure complementarity, the overexposed areas of the current reference frame are repaired using effective information from adjacent frames; multi-stream features are fused and the final HDR video frame is output.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed in a computer, causes the computer to perform the single-exposure Raw video HDR reconstruction method as described in any one of claims 1-6.

Images