Single-exposure Raw video HDR reconstruction method and system

Abstract

The invention relates to a single-exposure Raw video HDR reconstruction method and system, and belongs to the technical field of computer vision and image processing. The method comprises the following steps: acquiring a Raw video sequence in a single exposure Bayer format; a channel decomposition parallel alignment module is constructed, an input frame is decomposed into three parallel branches of a full channel flow, a red and blue channel flow and a double-green channel flow, and differential feature extraction and alignment are respectively performed by utilizing the physical characteristics that the red and blue channel is not easy to saturate in a highlight area and the green channel is high in signal-to-noise ratio; constructing an exposure complementary mask guide repair module, calculating an overexposure mask and dividing a time sequence complementary region, and repairing an overexposure region of the current frame by using normal exposure information of an adjacent frame in the complementary region; and utilizing a cross attention mechanism to fuse multi-stream features, and combining with mask guide color loss to reconstruct a high dynamic range video frame. According to the method, the problems of highlight overflow and dark noise are effectively solved under the condition that alternate exposure is not needed, ghosting artifacts are avoided, and the reconstruction quality of an HDR video is remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and image processing, and particularly relates to a single-exposure Raw video HDR reconstruction method and system. BACKGROUND

[0002] High dynamic range (HDR) imaging technology can record more rich brightness levels than low dynamic range (LDR) images, retain details from direct sunlight to deep shadows, and is widely used in film production, mobile photography, autonomous driving and virtual reality fields. However, due to the sensitivity and bit depth of the sensor, most consumer cameras often face the problems of highlight overflow (information loss) and dark noise amplification under single exposure.

[0003] In order to obtain an HDR video, the existing mainstream research mainly focuses on the "alternating exposure" technology, that is, frames of different exposure levels (such as long exposure and short exposure) are alternately shot in a time sequence, and then fused. However, this method has significant defects: first, it requires special shooting settings, resulting in a halving of the time resolution (frame rate); second, the inter-frame brightness difference is huge, which easily leads to registration difficulties in dynamic scenes, thereby producing ghosting artifacts.

[0004] In order to avoid the problem of alternating exposure, single-exposure HDR reconstruction has become a research hotspot. Early single-image HDR reconstruction mainly targets static images, ignoring the temporal correlation of videos, and most HDR methods are based on sRGB data processed by ISP. sRGB data has undergone non-linear mapping, white balance and quantization, irreversibly discarding key highlight and shadow information. In contrast, Raw sensor data has linear response and high bit depth (10-16 bits), retaining more rich radiation information. However, there is currently a lack of effective means to fully utilize the Bayer channel characteristics of Raw data and the temporal redundancy in single-exposure videos to restore details in extreme exposure areas.

[0005] Therefore, it is urgent to explore an innovative method that can fully utilize the linear characteristics of Raw data and the differences between Bayer channels, and achieve high-quality HDR video reconstruction without alternating exposure. SUMMARY

[0006] The application aims to at least solve the problems that the existing HDR video reconstruction method relies on alternating exposure, resulting in a decrease in frame rate and ghosting, and the information loss of sRGB data is serious, and provides a single-exposure Raw video HDR reconstruction method and system. The channel decomposition and parallel alignment strategy utilizes the physical characteristics of the Bayer array, and the exposure complementary mask guiding mechanism utilizes the temporal information to repair the overexposure area, realizing the reconstruction of high-quality HDR video from Raw video sequence under single-exposure conditions.

[0007] To achieve the above object, the technical scheme of the present application is: a single-exposure Raw video HDR reconstruction method, comprising:

[0008] Step S1, acquire a single-exposure Raw video sequence in Bayer format, select three consecutive frames as input, respectively a previous frame, a current reference frame and a next frame;

[0009] Step S2, construct a channel decomposition and parallel alignment module, decompose each input frame into three parallel branches of full channel flow, red and blue channel flow and double green channel flow; respectively extract features from the three branches, and align the features of adjacent frames to the current reference frame using optical flow; calculate the exposure weight mask, and differentially weight and fuse the red and blue channel flow and the double green channel flow, to finally obtain the features of the three aligned branches;

[0010] Step S3, construct an exposure complementary mask guided repair module, calculate the overexposure mask of each frame and perform motion compensation; according to the mask difference between the current reference frame and the adjacent frame, divide the spatial region into complementary region and double overexposure region; use the normal exposure information of the adjacent frame in the complementary region to repair the overexposure features of the current reference frame by weighted fusion;

[0011] Step S4, construct a fusion and reconstruction module, fuse the features obtained in step S3 and the aligned features of each branch extracted in step S2 using cross attention mechanism, and reconstruct the HDR Raw frame with high dynamic range through a multi-scale encoder-decoder network;

[0012] Step S5, construct a loss function containing a mask guided color loss, and train and optimize the network model using a pre-constructed RawHDRV dataset.

[0013] Further, step S1 specifically comprises:

[0014] acquire a single-exposure Raw video sequence , the data format is RGBG, and three consecutive frames are taken as input, respectively representing a previous frame, a current reference frame and a next frame; the input data is normalized in the Raw domain and the linear response characteristics are preserved.

[0015] Further, step S2 specifically comprises:

[0016] split each input frame in RGBG format into three parallel streams: (1) X stream: containing all four channels, used to maintain global context and structural consistency; (2) RB stream: containing red and blue channels, used for highlight detail recovery; (3) G stream: containing two green channels, used for dark noise reduction and detail preservation;

[0017] The optical flow network is used to calculate the optical flow from the adjacent frame to the current frame, and the adjacent frame features of each flow are warped to obtain the aligned features;

[0018] The cross-frame channel attention module is introduced to process the RB stream and the G stream, and pixel-level adaptive weights are used to weight the aligned features, and finally the aligned features of the three branches are obtained.

[0019] Further, the pixel-level adaptive weights are calculated according to the overexposure mask , the underexposure mask and the normal exposure mask : for the RB stream, a higher weight is given to the overexposure area , so as to utilize the characteristic that the red and blue channels are not easy to saturate in the highlight area; for the G stream, a higher weight is given to the underexposure area , so as to utilize the characteristic that the green channel has high signal-to-noise ratio.

[0020] Further, step S3 specifically comprises:

[0021] Calculate the overexposure mask of each frame, and transform the mask of the adjacent frame to the coordinate system of the current reference frame using the optical flow; compare the mask of the current reference frame with the transformed mask of the adjacent frame to divide the complementary area and the double overexposure area; calculate the fusion weight based on the complementary area , enhance the weight of the adjacent frame feature in the complementary area, and repair the overexposure feature of the current reference frame; reduce the weight of the features of the current reference frame and the adjacent frame in the double overexposure area to suppress error propagation.

[0022] Further, the complementary area is defined as: the area where the current reference frame is in the overexposure state, but the adjacent frame is in the normal exposure state at the same position.

[0023] Further, the double overexposure area is defined as: the area where the current reference frame and the adjacent frame are both in the overexposure state.

[0024] Further, in step S4, in the multi-scale encoder-decoder network, the encoder adopts a three-layer down-sampling path, and the decoder adopts a symmetric up-sampling path and introduces cross-layer skip connection.

[0025] Further, in step S5, the loss function includes an L1 loss, a log domain L2 loss, and a mask guided color loss; the mask guided color loss includes: (1) a mask normalized L1 term: only the L1 distance between the predicted result and the true value in the overexposure mask area is calculated, and normalization is performed using the mask area, so as to force the network to pay attention to highlight recovery; (2) a gradient smoothing term: the gradient of the predicted result in the horizontal and vertical directions is calculated, the spatial continuity of the highlight boundary is constrained, and the artifact is reduced.

[0026] The application further provides a single-exposure Raw video HDR reconstruction system for performing the single-exposure Raw video HDR reconstruction method.

[0027] A data acquisition module is configured to acquire a Raw video sequence captured in a single-exposure mode.

[0028] A data processing module is configured to perform channel decomposition and parallel alignment, extract and align features of full-channel streams, red-blue channel streams and double-green channel streams, repair overexposure areas of a current reference frame by using effective information of adjacent frames according to exposure complementarity, and fuse multi-stream features and output a final HDR video frame.

[0029] The application further provides a computer readable storage medium having a computer program stored thereon, and the computer program is configured to enable a computer to perform the single-exposure Raw video HDR reconstruction method when the computer program is executed in the computer.

[0030] The application further provides a computing device including a memory and a processor, and the memory is configured to store executable codes, and the processor is configured to execute the executable codes to implement the single-exposure Raw video HDR reconstruction method.

[0031] Compared with the prior art, the application has the following beneficial effects:

[0032] (1) The application proposes a parallel processing architecture based on channel decomposition, and for the first time, the physical characteristic differences (i.e., the red-blue channel is beneficial to highlight, and the green channel is beneficial to dark part) of different color channels in a Raw image Bayer array in noise and saturation are systematically utilized, so that the detail recovery ability under extreme exposure conditions is significantly improved.

[0033] (2) The application introduces an exposure complementary mask guided mechanism, and ingeniously utilizes the temporal redundancy in a video sequence, i.e., the "position-exposure" change caused by object or camera motion is utilized to borrow unsaturated pixel information from adjacent frames to repair the overexposure area of a current frame, so that the problem of complete loss of highlight information under single exposure is effectively solved.

[0034] (3) The application designs a mask-guided color loss function, which specifically constrains the chroma consistency and gradient smoothness of the highlight area, effectively eliminating the common color deviation and artifact problems in highlight reconstruction;

[0035] (4) The application breaks away from the dependence on alternate exposure shooting, avoids the frame rate loss and ghosting problem in dynamic scenes, and provides a feasible solution for mobile real-time high-quality HDR video acquisition. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a single-exposure Raw video HDR reconstruction method flowchart provided by the application;

[0037] Figure 2 is the overall architecture diagram of the RawHDRV network in the application, which shows the connection relationship of channel decomposition, parallel alignment and mask-guided repair;

[0038] Figure 3 is a structural schematic diagram of the channel decomposition and parallel alignment (CDPA) module in the application;

[0039] Figure 4 is a principle schematic diagram of the exposure complementary mask-guided repair (MGR) module in the application, which shows the division logic of the complementary area;

[0040] Figure 5 is the result diagram of reconstructing the HDR image on the RawHDRV dataset by using the method of the application, (a) is a continuous three-frame low dynamic range (LDR) video sequence of the input model, and (b) is a high dynamic range (HDR) image corresponding to the center frame after being processed by the model. DETAILED DESCRIPTION

[0041] The technical solutions of the application will be specifically described below with reference to the drawings.

[0042] The application provides a single-exposure Raw video HDR reconstruction method, which comprises:

[0043] Step S1, acquire a single-exposure Bayer format Raw video sequence, select three continuous frames as input, which are the previous frame, the current reference frame and the next frame;

[0044] Step S2, construct a channel decomposition and parallel alignment module, decompose each input frame into three parallel branches of full channel flow, red-blue channel flow and double green channel flow; respectively extract features from the three branches, and align the features of adjacent frames to the current reference frame by using optical flow; calculate the exposure weight mask, and differentially weight and fuse the red-blue channel flow and the double green channel flow, to finally obtain the aligned features of the three branches;

[0045] Step S3: Construct an exposure complementary mask-guided repair module, calculate the overexposure mask of each frame and perform motion compensation; divide the spatial region into complementary region and double overexposure region according to the mask difference between the current reference frame and adjacent frames; use the normal exposure information of adjacent frames in the complementary region to repair the overexposure features of the current reference frame through weighted fusion.

[0046] Step S4: Construct a fusion and reconstruction module. Use the cross-attention mechanism to fuse the features obtained in step S3 and the features after branch alignment extracted in step S2. Reconstruct the high dynamic range HDR raw frame through a multi-scale encoder-decoder network.

[0047] Step S5: Construct a loss function that includes mask-guided color loss, and train and optimize the network model using the pre-built RawHDRV dataset.

[0048] The following are specific implementation examples of the present invention.

[0049] Example 1:

[0050] This embodiment discloses a single-exposure Raw video HDR reconstruction method. This method directly processes Raw sensor data and utilizes Bayer channel characteristics and temporal complementarity to recover high dynamic range. See appendix. Figure 1 The above method includes the following steps:

[0051] Step S1: Obtain the target Raw video sequence and preprocess it.

[0052] Acquiring a single-exposure raw video sequence The data format is RGBG (four-channel Bayer mosaic). This invention uses three consecutive frames... As input, the aim is to reconstruct the intermediate frame. Corresponding HDR image The input data is normalized in the Raw domain while preserving linear response characteristics.

[0053] Step S2: Construct a Channel Decomposition Parallel Alignment (CDPA) module to extract and align features.

[0054] like Figure 2 and Figure 3 As shown, considering that green pixels in Bayer sensors have higher photon capture efficiency and signal-to-noise ratio (suitable for dark areas), while red and blue pixels are less prone to saturation in bright areas (suitable for highlights), this embodiment designs a three-way parallel channel (branch) stream:

[0055] (1) X-stream (full channel): Input complete 4-channel data and extract features. It is used to maintain global structure and context information;

[0056] (2) RB stream (Red-Blue channel): only input Red and Blue channel, extract features , focus on highlight recovery;

[0057] (3) G (G1G2) stream (double green channel): input two Green channels, extract features ( ), focus on dark noise reduction.

[0058] Calculate the optical flow from the adjacent frame to the middle frame using the pre-trained optical flow network (such as SPyNet) , and perform Warping operation Warp on the adjacent frame features of each stream to obtain the aligned features:

[0059]

[0060] To eliminate the registration error caused by the optical flow residual, the application introduces a cross-frame channel cross-attention module to perform feature fusion between the aligned adjacent frame features and the center frame features:

[0061]

[0062] wherein represents the complementary features weighted by attention CA. This operation is performed independently in the three channel streams to preserve the channel-level characteristics and local exposure differences.

[0063] To further utilize the channel characteristics, an adaptive weighting based on the exposure mask is introduced. First, calculate the overexposure mask (e.g. pixel value > 0.95) and underexposure mask (e.g. pixel value < 0.05), and normal exposure mask .

[0064] Apply differential weights to the RB stream and G stream, the process is represented as:

[0065]

[0066]

[0067] wherein . Then output the weighted channel features:

[0068]

[0069] wherein is the element-wise multiplication. This design embodies the principle of differential weighting of channels: in the overexposure area, enhance the RB stream to recover highlights, in the underexposure area, enhance the G stream to enhance dark structures, and in the normal area, balance the contributions of the three streams.

[0070] Step S3, construct an exposure complementary mask guided repair module (MGR) to repair overexposed regions.

[0071] As shown in Figure 4 , due to the motion of the camera or the photographed object, the current (intermediate) frame has pixels that are saturated (overexposed), while the corresponding locations in the adjacent frames or can not be saturated. This step takes advantage of this characteristic for repair.

[0072] First, calculate the overexposure mask for each frame and warp the adjacent frame mask to the center frame coordinate system through the optical flow field. Taking forward propagation as an example, for the overexposure mask of frame :

[0073]

[0074] By comparing the current frame mask with the propagated mask , the space is divided into three types of regions:

[0075] where denotes the element-wise product, , is the binary mask, is the binary operation, is the threshold, set to 0.5.

[0076] (1) Complementary region (C): the current frame is overexposed, but the adjacent frame is normal. This is the most valuable recovery region.

[0077] (2) Double overexposed region (D): both frames are overexposed. It needs to be suppressed to prevent the introduction of artifacts.

[0078] Based on the above division, the fusion weights and are calculated, and the process is represented as:

[0079]

[0080]

[0081] where (normal frame enhancement factor) and ​​(overexposure frame suppression factor). This strategy greatly enhances the contribution of normal frames in complementary regions, significantly suppresses the interference of overexposure frames, and simultaneously attenuates both in double overexposure regions to avoid error propagation. The above weighting operation acts on the feature pair after the warping operation , output enhanced features , and input into the cross-frame cross-attention module for fusion:

[0082]

[0083] The whole process is independently executed in the forward ( ) and backward ( ) directions, and finally two propagation features are obtained at the center frame, which are spliced with the multi-channel features and then convolved by 3x3 to integrate:

[0084]

[0085] where represents channel-level splicing. Then input the reconstruction module to ensure the maximum utilization of temporal context information.

[0086] Step S4, fusion and reconstruction.

[0087] The multi-channel and temporal enhancement features are input into the multi-scale encoder-decoder structure for feature integration and high-level semantic reconstruction. The encoder adopts a three-layer down-sampling path to capture global context information at different scales: from the first to the third level, [2, 3, 4] TransformerBlock stacks are configured respectively; the decoder adopts a symmetric up-sampling path and introduces cross-layer skip connections to maintain local structural details. Finally, the output features are reconstructed into HDRRaw frames through the refining module and convolution layer, and are fused with the center frame through residual connection:

[0088]

[0089] where represents the combination of the encoder-decoder and the refining module. This design ensures that the model simultaneously considers brightness range expansion and texture detail fidelity during reconstruction.

[0090] Step S5, model training and loss function calculation.

[0091] A combined loss function is used in the training stage of the present application:

[0092]

[0093] where the mask-weighted color loss weight λ is set to 0.3. The L1 loss Guarantee the consistency of overall brightness; L2 loss in logarithmic domain Calculate in logarithmic domain, balance the weight of light and dark parts.

[0094] The key innovation is the mask-guided color loss The calculation formula is:

[0095]

[0096] Among them, and respectively represent the network predicted HDR frame and the corresponding real HDR label; and respectively predict the gradient of the HDR in the horizontal direction and the vertical direction, which is used to describe the local continuity of brightness and color change. Symbol represents the pixel-by-point inner product operation, is norm, represents the expectation of the spatial dimension; , , Ensure numerical stability.

[0097] The first term uses the overexposure mask Only calculate the normalized error of the highlight area, force the network to focus on restoring the highlight color; The second term is the gradient smoothing term, which prevents the high light boundary from appearing fault or artifact.

[0098] The embodiment also provides a single-exposure Raw video HDR reconstruction system based on the above method, comprising:

[0099] A data acquisition module configured to connect an image sensor and read a Raw video data stream in Bayer format;

[0100] A data processing module, which is internally provided with parallel convolutional neural network branches for processing full-band, long-band (red and blue) and medium-band (green) signals respectively; contains a logical operation unit for generating a mask, optical flow transformation and regional logical judgment, and performs weighted operation; contains a Transformer calculation unit for performing deep fusion of features and image reconstruction.

[0101] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, when the computer program is executed in a computer, the computer executes the method described in any one of the embodiments.

[0102] The embodiment of the present application provides a kind of computing device, including memory and processor, the memory has executable code, the processor executes the executable code, and the method of any one described in the implementation is realized.

[0103] Wherein, memory can include high-speed random access memory (RAM), possibly also including non-volatile memory (Non-volatile Memory). Processor can be an integrated circuit chip, including central processing unit (CPU), graphics processor (GPU), digital signal processor (DSP), application specific integrated circuit (ASIC) or field programmable gate array (FPGA) and the like.

[0104] Finally, it should be noted that: the above-described embodiments, only for the specific embodiments of the present application, to illustrate the technical solutions of the present application, rather than limit it, the protection scope of the present application is not limited to this, although the foregoing detailed description of the present application is described, those skilled in the art should understand: any familiar with the technical field of the technical person in the technical range disclosed by the present application, it still can be modified or easily thought of change to the technical solution recorded in the foregoing examples, or equivalent replacement to part of technical features;And these modifications, changes or replacement, do not make the corresponding technical solution of the essence deviate from the spirit and scope of the technical solution of the embodiment of the present application, all should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be the protection scope of the claims.

Claims

1. A method for HDR reconstruction of single-exposure Raw video, characterized in that, include: Step S1: Obtain a single-exposure Raw video sequence, selecting three consecutive frames as input, namely the previous frame, the current reference frame, and the next frame; Step S2: Construct a channel decomposition parallel alignment module to decompose each input frame into three parallel branches: full channel stream, red-blue channel stream, and dual green channel stream; Feature extraction is performed on the three branches respectively, and optical flow is used to align the features of adjacent frames to the current reference frame; by calculating the exposure weight mask, differential weighted fusion is performed on the red-blue channel stream and the dual green channel stream to finally obtain the features after the three branches are aligned. Step S3: Construct an exposure complementary mask-guided repair module, calculate the overexposure mask for each frame and perform motion compensation; based on the mask difference between the current reference frame and adjacent frames, divide the spatial region into complementary regions and double overexposure regions. By utilizing the normal exposure information of adjacent frames in complementary regions, the overexposure characteristics of the current reference frame are repaired through weighted fusion. Step S4: Construct a fusion and reconstruction module. Use the cross-attention mechanism to fuse the features obtained in step S3 and the features after branch alignment extracted in step S2. Reconstruct high dynamic range (HDRRaw) frames through a multi-scale encoder-decoder network. Step S5: Construct a loss function that includes mask-guided color loss, and train and optimize the network model using the pre-built RawHDRV dataset.

2. The method for HDR reconstruction of single-exposure Raw video according to claim 1, characterized in that, Step S1 specifically includes: Acquiring a single-exposure raw video sequence The data format is RGBG, in three consecutive frames. As input, These represent the previous frame, the current reference frame, and the next frame, respectively; the input data is normalized in the Raw domain while preserving linear response characteristics.

3. The method for HDR reconstruction of single-exposure Raw video according to claim 1, characterized in that, Step S2 specifically includes: Each input frame in RGBG format is split into three parallel streams: (1) X stream: containing all four channels; (2) RB stream: containing the red and blue channels; (3) G stream: containing two green channels; The optical flow from adjacent frames to the current frame is calculated using an optical flow network, and the adjacent frame features of each flow are warped to obtain aligned features. A cross-frame channel attention module is introduced to process the RB stream and G stream, and pixel-level adaptive weights are used to weight the aligned features, finally obtaining the aligned features of the three branches.

4. The method for HDR reconstruction of single-exposure Raw video according to claim 3, characterized in that, The pixel-level adaptive weights are based on the overexposure mask. Underexposure mask and normal exposure mask Calculated.

5. The method for HDR reconstruction of single-exposure Raw video according to claim 1, characterized in that, Step S3 specifically includes: Calculate the overexposure mask for each frame. The system uses optical flow to transform the masks of adjacent frames to the coordinate system of the current reference frame; it then compares the mask of the current reference frame with the transformed masks of adjacent frames to delineate complementary regions. and double overexposure regions; based on complementary regions Calculate the fusion weights, enhance the weights of features from adjacent frames in complementary regions, and correct the overexposure features of the current reference frame; reduce the weights of features from the current reference frame and adjacent frames in double overexposure regions.

6. The method for HDR reconstruction of single-exposure Raw video according to claim 5, characterized in that, The complementary region Defined as: the area where the current reference frame is overexposed, but adjacent frames are in the same position and are in the normal exposure state.

7. The method for HDR reconstruction of single-exposure Raw video according to claim 1, characterized in that, In step S4, in the multi-scale encoder-decoder network, the encoder adopts a three-layer downsampling path, and the decoder adopts a symmetrical upsampling path and introduces cross-layer skip connections.

8. The method for HDR reconstruction of single-exposure Raw video according to claim 1, characterized in that, In step S5, the loss function includes L1 loss, logarithmic domain L2 loss and mask-guided color loss; the mask-guided color loss includes: (1) mask normalization L1 term: calculate the L1 distance between the prediction result and the true value only in the overexposed mask area, and normalize it using the mask area; (2) gradient smoothing term: calculate the gradient of the prediction result in the horizontal and vertical directions.

9. A single-exposure Raw video HDR reconstruction system that performs the single-exposure Raw video HDR reconstruction method as described in any one of claims 1-8, characterized in that, include: The data acquisition module is responsible for acquiring the Raw video sequence captured in single-exposure mode; The data processing module is responsible for performing channel decomposition and parallel alignment, extracting and analyzing the features of the complete channel stream, red-blue channel stream, and dual-green channel stream. By leveraging exposure complementarity, the overexposed areas of the current reference frame are repaired using effective information from adjacent frames; multi-stream features are fused and the final HDR video frame is output.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed in a computer, causes the computer to perform the single-exposure Raw video HDR reconstruction method as described in any one of claims 1-8.

Images