A method for reconstructing HDR video based on semi-supervised learning

Abstract

A method for reconstructing HDR video based on semi-supervised learning 1) prepares a small amount of labeled sample data; 2) two kinds of samples are respectively input into an optical flow estimation network to obtain aligned LDR frames; 3) the aligned LDR frames are respectively input into a merging network, a feature sequence is obtained through a mixed hollow convolution module, and the key area is given more weight through an attention module to correct the feature sequence; 4) the corrected feature sequence is concatenated and input into an encoder-decoder, the encoder-decoder obtains an output result by calculating the weight of each pixel of all input frames, and the final HDR frame is reconstructed through the output result of the average weighted value of the input frame; 5) the processes of the two kinds of samples can be regarded as two branches, namely a supervised branch and an unsupervised branch, and semi-supervised learning is performed by calculating and comparing the training loss functions of the two branches.

Description

Technical Field

[0001] This invention belongs to the field of HDR (High Dynamic Range) video reconstruction and is a method for reconstructing HDR videos based on semi-supervised learning. It is a semi-supervised learning framework based on a merge network with attention and hybrid dilated convolution to reconstruct high dynamic range videos with a small number of samples, especially in the direction of reconstructing HDR videos using a small amount of labeled training data (i.e., a small number of HDR videos and their corresponding LDR (Low Dynamic Range) input videos) and unlabeled LDR videos. Background Technology

[0002] Compared to the dynamic range perceived by the human eye, most standard digital cameras can only capture a limited dynamic range within a scene. This results in underexposed or overexposed areas in the captured LDR video, leading to a loss of visual detail. HDR video reconstruction technology aims to solve this problem without the need for expensive specialized sensors, generating HDR videos that more closely resemble the image perceived by the human eye. While significant progress has been made in HDR imaging technology recently, HDR video reconstruction has received relatively little attention.

[0003] Currently, there are two main methods for reconstructing HDR videos: a conventional method that aligns and merges adjacent frames and reference frames from alternating exposure LDR videos to generate HDR frames, and a deep learning method that merges LDR frames based on a variable alignment neural network to generate and refine HDR frames. However, both require large amounts of HDR video datasets and lengthy training times. Furthermore, the conventional method suffers from alignment and merging errors due to scene motion and missing information in underexposed areas, leading to the incorrect generation of "ghosting" and noise. Summary of the Invention

[0004] The main problems addressed by this invention are: proposing a semi-supervised learning framework for reconstructing HDR videos, achieving better results with small sample sizes; and utilizing a merge network based on attention and hybrid dilated convolutions to generate realistic images and remove ghosting and noise. This involves a semi-supervised learning framework based on attention and hybrid dilated convolutions for reconstructing high dynamic range videos with small sample sizes. The purpose of this invention is to address the problem that reconstructing HDR videos requires a significant amount of time to prepare and train datasets, resulting in poor quality reconstructed HDR videos (such as ghosting and unrestored details). In the absence of large-scale real-world HDR video datasets, this invention aims to reconstruct HDR videos that more closely resemble human visual perception.

[0005] The technical solution of this invention is a method for reconstructing HDR videos based on semi-supervised learning, comprising: 1) an overall framework based on semi-supervised learning; 2) a merge network based on attention and hybrid dilated convolution; and 3) a loss function based on semi-supervised learning.

[0006] The process is as follows: 1) Prepare a small amount of labeled sample data, namely HDR videos with ground truth and their corresponding LDR input videos, and unlabeled sample data, namely LDR videos without ground truth, and perform CRF (Camera Response Function) replacement on the sample data to ensure a uniform format; 2) Input the two types of samples into the optical flow estimation network respectively to obtain aligned LDR frames; 3) Input the aligned LDR frames into the merging network respectively, obtain feature sequences through the hybrid dilated convolution module, and then give more weight to key regions through the attention module to correct the feature sequences; 4) The corrected feature sequences will be concatenated and input into the encoder-decoder. The encoder-decoder obtains the output result by calculating the weight of each pixel in all input frames, and reconstructs the final HDR frame through the output result of the average weighted value of the input frames; 5) The process for the two types of samples can be regarded as two branches, namely supervised branch and unsupervised branch. Semi-supervised learning is performed by calculating and comparing the training loss functions under the two branches.

[0007] High-quality HDR video reconstruction using small samples; semi-supervised learning through parallel supervised and unsupervised branches; and three features avoided: ghosting, detail errors, and noise, through a merge network based on attention and hybrid dilated convolutions.

[0008] L i This is the correct LDR input frame after being replaced by CRF. For CRF, γ is a parameter of the gamma curve, set to 2.2. The CRF replacement process is defined as follows:

[0009]

[0010] Within the overall framework of semi-supervised learning, the semi-supervised learning framework utilizes LDR videos covering different real-world scenes to reconstruct HDR videos.

[0011] With N l Each labeled sample is an HDR video with ground truth and its corresponding LDR video input, N l Small sample dataset D (no more than 5) l With N u A dataset D of unlabeled samples (LDR videos without ground truth) u As input to the neural network, the neural network f in the overall framework of semi-supervised learning contains a supervised branch f. s and an unsupervised branch f uThe two inputs, labeled LDR video input and unlabeled LDR video input, are processed separately. During training, both video input branches have the same weight throughout the neural network. l With D u The samples are simultaneously used as inputs into the two branches; after the HDR video is reconstructed in the two video input branches respectively, the supervised loss function and the unsupervised loss function are calculated and used to update f.

[0012] Merge network based on attention and hybrid dilated convolution;

[0013] Hybrid dilated convolution modules are used for feature extraction of LDR frames, with input I... i Given I x , Feature sequence, m is the porosity, h m (·) represents the corresponding dilated convolutional layer, Concat is the concatenation operation, h(·) is the convolutional layer that reduces the size of the concatenated features, and the extraction process is defined as:

[0014]

[0015]

[0016] After the hybrid dilated convolution module, the feature sequence The features are extracted and fed into the attention module; the attention module is used to correct the feature sequence other than the reference frame; the attention module contains only two convolutional layers, and each attention module requires two inputs, namely features from the neighboring frame and the reference frame; after sigmoid activation, the value of the attention map is restricted to the range of [0,1];

[0017] The corrected feature sequences are concatenated and used as input to the encoder-decoder. The encoder-decoder calculates the weight of each pixel in all input frames to obtain the output result. The HDR frame is reconstructed from the output result of the average weighted value of the input frames.

[0018] Semi-supervised learning loss function:

[0019] The supervised loss function is The unsupervised loss function is β is the unsupervised loss function The weights, therefore the total loss function for:

[0020]

[0021] and The weighted sum is used as the final loss function, and the weights are updated by backpropagation using this loss function each time the network takes samples for training. As a supervised loss function, it plays a strong constraining role. As an unsupervised loss function, it can be seen as a supplement, indirectly increasing the number of samples participating in training.

[0022] c) Supervised loss function

[0023] Supervised loss function From the absolute value loss function With perceptual loss function Composition, α is Weights:

[0024]

[0025] To ensure that HDR frames generated by supervised branches are Approaching HDR Ground Truth An absolute value loss function was introduced. and They are respectively for and A tone-mapped image, defined as:

[0026]

[0027] Introducing a perceptual loss function pre-trained in the VGG-16 network and For k from VGG-16 th The features extracted by the layer are defined as follows:

[0028]

[0029] d) Unsupervised loss function

[0030] Unsupervised loss function From the total variation loss function Consistency loss function Alignment loss function The composition, where θ and λ are the corresponding weights:

[0031]

[0032] Total variational loss function This can improve the spatial smoothness of unsupervised branching in generating HDR frames, preserving overall structure and detail while removing noise. and These are the horizontal and vertical difference operation matrices, respectively. The video output for the unsupervised branch is defined as follows:

[0033]

[0034] Since the HDR frames generated by the network have a similar structure to the reference LDR frames, a consistency loss function is introduced. Input video for the unsupervised branch of LDR. For the corresponding exposure level, l ssim The difference in SSIM calculation is a sensory quantification of image quality; after exposure adjustment, a gamma curve with γ = 2.2 is used to transfer the HDR domain frame to the LDR domain. Clip is a function that keeps the output value within [0,1]. The consistency loss function provides weak supervision to restore the correct shape and detail of the reference frame, improving the visual experience of the generated HDR frame. It is defined as:

[0035]

[0036] An alignment loss function was introduced. As a constraint to improve the results of the optical flow estimation network, for two alternately exposed LDR frames, the adjacent frames of the LDR reference frame have the same exposure, so the alignment loss function is defined as the shape difference between the two. and Aligned LDR frames and Grayscale image:

[0037]

[0038] Furthermore, the steps for reconstructing HDR video using a small number of labeled samples (HDR video and its corresponding LDR input video) and unlabeled samples (LDR video) are as follows:

[0039] Preprocess the sample data and use CRF (Camera Response Function) replacement to unify the format of LDR input videos captured by different devices; L i This is the correct LDR input frame after being replaced by CRF. For CRF, γ is a parameter of the gamma curve, set to 2.2. The CRF replacement process is defined as follows:

[0040]

[0041] The LDR input frame is fed into the optical flow estimation network, which is an improved version of SPyNet. It consists of four consecutive convolutional layers. SPyNet is a high-performance deep learning-based optical flow estimation method that can achieve good results even with video frames of different exposure levels as input. Therefore, it is used as an auxiliary tool for optical flow estimation. Re-exposure is used to improve the performance of optical flow estimation. Afterwards, inference calculations are performed in the four consecutive convolutional layers of the optical flow estimation network.

[0042] Reference frame L i It will be based on adjacent exposure t i+1 To perform a reexposure, Clip is a function that keeps the output value within the range of [0,1]. γ is a parameter of the gamma curve, set to 2.2. The reexposure process is defined as follows:

[0043]

[0044] After reexposure and inference, the optical flow estimation map generated by the optical flow estimation network is used to transform adjacent frames, resulting in five aligned LDR frames. Since the information contained in three consecutive frames is limited, the final generated HDR frame must be aligned with the intermediate frame in texture. Therefore, aligned neighboring frames are needed to supplement the information, hence five LDR frames are required. These five frames are concatenated and mapped to the corresponding HDR domain, and gamma correction along the channel dimension is used to generate the corresponding LDR input frame I. i =(I i-1,i ,I i-1 ,I i ,I i+1 ,I i+1 ,i);

[0045] The LDR input frames generated by the optical flow estimation network are fed into the merging network; after feature extraction by the hybrid dilated convolution module, the feature sequences are fed into the attention module for correction.

[0046] The beneficial effects of this invention are: reconstructing HDR videos with a small sample size, without spending a lot of time, avoiding the generation of "ghosting" and noise, and generating higher quality HDR videos.

[0047] The features of this invention are: 1. Semi-supervised learning is performed in parallel with supervised and unsupervised branches to reconstruct HDR videos with small samples, without having to spend a lot of time collecting and processing huge datasets; 2. A higher quality HDR video is generated by a merging network based on attention and hybrid dilated convolutions, avoiding the generation of "ghosting", detail errors and noise. Attached Figure Description

[0048] Figure 1 This is a workflow view of the present invention.

[0049] Figure 2 This is a schematic diagram of the merge network.

[0050] Figure 3 This is a schematic diagram of a hybrid dilated convolution module.

[0051] Figure 4 This is a schematic diagram of the attention module.

[0052] Figure 5 This section presents a qualitative comparison of LDR reference frames under exposure conditions on two sequences within the same synthetic evaluation dataset. Each large row represents the different results obtained by one method under four test cases across two sequences, totaling seven rows: Kalantari19, Kalantari19-R, Chen21, Chen21-R, Prabhakar21, this method, and the actual output image. Each large column represents the different results obtained by one test case under different methods, totaling four columns. The upper half of each block represents the overall image, and the lower half represents overexposed and underexposed slices of the image above.

[0053] Figure 6 This demonstrates a qualitative comparison of visual effects on the same synthetic evaluation dataset. It is divided into six sections in two rows and three columns, from top to bottom and left to right: Kalantari19, Chen21, Prabhakar21, Kalantari19-R, and our method; each section has two overall images on the right and slices of those two images on the left.

[0054] Figure 7 This section presents a qualitative comparison of the visual effects on the Kalantari13 dataset. It is divided into six parts in two rows and three columns, from top to bottom and left to right: Kalantari19, Chen21, Prabhakar21, Kalantari19-R, and our method; the right side of each part shows two overall images, and the left side shows slices of the two images. Detailed Implementation

[0055] The key technologies involved in this invention are: an overall framework based on semi-supervised learning, a merge network based on attention and hybrid dilated convolution, and a semi-supervised learning loss function. These will be explained below:

[0056] The overall framework based on semi-supervised learning:

[0057] Existing deep learning-based HDR video reconstruction algorithms are developed within a supervised learning framework, which is limited by the requirement for a large amount of labeled sample data. Furthermore, due to the scarcity of real-world HDR video datasets, the HDR videos generated by this method are often used as substitutes for real-world HDR videos in training. In contrast, LDR videos covering different real-world scenes, while lacking ground truth, are more readily available, and there is currently no research utilizing them for HDR video reconstruction. Semi-supervised learning frameworks leverage the potential of these unlabeled samples to achieve HDR video reconstruction.

[0058] Specifically, with N l N labeled samples (HDR videos with ground truth and their corresponding LDR video inputs, N) l Small sample dataset D (no more than 5) l With N u A dataset D of unlabeled samples (LDR videos without ground truth) u As input to the neural network, the entire neural network f contains a supervised branch f. s and an unsupervised branch f u The two inputs are processed separately, and the two branches have the same weights throughout the neural network during training. l With D u The samples are simultaneously input into both branches. After reconstructing the HDR video in each branch, supervised and unsupervised loss functions are calculated and used to update f. The specific calculation method of the loss function is as follows.

[0059] Figure 1 This is also a schematic diagram of the overall framework based on semi-supervised learning.

[0060] Merge networks based on attention and hybrid dilated convolutions:

[0061] Figure 2 This is a schematic diagram of the merge network.

[0062] Due to object motion and underexposure, traditional convolution cannot extract sufficient local region information from alternating exposure frames. Therefore, this technique employs dilated convolution. Compared to traditional convolution, dilated convolution reduces the loss of spatial features and obtains similar features in distant areas of the image with lower computational complexity. To avoid the grid effect caused by extending the sampling interval and losing original information, this technique uses hybrid dilated convolution to obtain additional spatial information.

[0063] Hybrid dilated convolution modules are used for feature extraction, with input I... i Given I xFor example, Here is the characteristic sequence, m is the porosity, and h is the number of holes. m (·) represents the corresponding dilated convolutional layer, Concat is the concatenation operation, and h(·) is the convolutional layer that reduces the size of the concatenated features. This extraction process is defined as:

[0064]

[0065]

[0066] Figure 3 This is a schematic diagram of a hybrid dilated convolution module. Following the hybrid dilated convolution module, the feature sequence F... h The features are extracted and fed into the attention module. Since the attention module is effective in suppressing "ghosting" and erroneous details caused by the introduction of adjacent non-reference frames, it is used to correct feature sequences other than the reference frame. The attention module contains only two convolutional layers, and each attention module requires two inputs: features from adjacent frames and features from the reference frame. After sigmoid activation, the values ​​of the attention map are restricted to the range [0,1].

[0067] Figure 4 This is a schematic diagram of the attention module. The corrected feature sequence is concatenated and used as input to the encoder-decoder. The encoder-decoder obtains the output by calculating the weight of each pixel in all input frames. The HDR frame is reconstructed from the output of the average weighted value of the input frames.

[0068] Semi-supervised learning loss function:

[0069] The supervised loss function is The unsupervised loss function is β is the unsupervised loss function The weights, therefore the total loss function for:

[0070]

[0071] and The weighted sum is used as the final loss function, and the weights are updated by backpropagation using this loss function each time the network takes samples for training. As a supervised loss function, it plays a strong constraining role. As an unsupervised loss function, it can be seen as a supplement, indirectly increasing the number of samples participating in training.

[0072] e) Supervised loss function

[0073] Supervised loss function From the absolute value loss function With perceptual loss function Composition, α is Weights:

[0074]

[0075] To ensure that HDR frames generated by supervised branches are Approaching HDR Ground Truth An absolute value loss function was introduced. and They are respectively for and A tone-mapped image, defined as:

[0076]

[0077] Introducing a perceptual loss function pre-trained in the VGG-16 network and For k from VGG-16 th The features extracted by the layer are defined as follows:

[0078]

[0079] f) Unsupervised loss function

[0080] Unsupervised loss function From the total variation loss function Consistency loss function Alignment loss function The composition, where θ and λ are the corresponding weights:

[0081]

[0082] Total variational loss function This can improve the spatial smoothness of unsupervised branching in generating HDR frames, preserving overall structure and detail while removing noise. and These are the horizontal and vertical difference operation matrices, respectively. The video output for the unsupervised branch is defined as follows:

[0083]

[0084] Since the HDR frames generated by the network have a similar structure to the reference LDR frames, a consistency loss function is introduced. Input video for the unsupervised branch of LDR. For the corresponding exposure level, l ssimThe difference in SSIM calculation is a sensory quantification of image quality; after exposure adjustment, a gamma curve with γ = 2.2 is used to transfer the HDR domain frame to the LDR domain. Clip is a function that keeps the output value within [0,1]. The consistency loss function provides weak supervision to restore the correct shape and detail of the reference frame, improving the visual experience of the generated HDR frame. It is defined as:

[0085]

[0086] An alignment loss function was introduced. As a constraint to improve the results of the optical flow estimation network, for two alternately exposed LDR frames, the adjacent frames of the LDR reference frame have the same exposure, so the alignment loss function is defined as the shape difference between the two. and Aligned LDR frames and Grayscale image:

[0087]

[0088] Usage Example

[0089] The steps to reconstruct HDR video from a small number of labeled samples (HDR video and its corresponding LDR input video) and unlabeled samples (LDR video) are as follows:

[0090] 1) Preprocess the sample data and use CRF replacement to unify the format of LDR input videos captured by different devices. i This is the correct LDR input frame after being replaced by CRF. Here, γ is the parameter of the CRF (Continuous Radiographic Facility), set to 2.2. The CRF replacement process is defined as follows:

[0091]

[0092] 2) The LDR input frame is fed into the optical flow estimation network, an improved version of SPyNet, which consists of four consecutive convolutional layers. SPyNet, as a high-performance deep learning-based optical flow estimation method, achieves decent results even with video frames of varying exposure levels as input; therefore, it is used as an auxiliary tool for optical flow estimation. Re-exposure is used to improve the performance of optical flow estimation, and inference calculations are then performed within the four consecutive convolutional layers of the optical flow estimation network. Reference frame L i It will be based on adjacent exposure t i+1 To perform a reexposure, Clip is a function that keeps the output value within the range [0,1]. γ is a parameter of the gamma curve, set to 2.2. The reexposure process is defined as follows:

[0093]

[0094] After reexposure and inference, the optical flow estimation map generated by the optical flow estimation network is used to transform adjacent frames, resulting in five aligned LDR frames. Since the information contained in three consecutive frames is limited, the final generated HDR frame must be aligned with the intermediate frame in texture. Therefore, aligned neighboring frames are needed to supplement the information, hence five LDR frames are required. These five frames are concatenated and mapped to the corresponding HDR domain, and gamma correction along the channel dimension is used to generate the corresponding LDR input frame I. i =(I i-1,i ,I i-1 ,I i ,I i+1 ,I i+1,i ).

[0095] 3) The LDR input frames generated by the optical flow estimation network are fed into the merging network. After feature extraction by the hybrid dilated convolution module, the feature sequences are fed into the attention module for correction. The corrected feature sequences are concatenated and used as input to the encoder-decoder. The encoder-decoder calculates the weight of each pixel in all input frames to obtain the output result. The HDR frame is reconstructed from the output result of the average weighted value of the input frames.

[0096] 4) Combine supervised and unsupervised loss functions and perform semi-supervised learning.

[0097] Sample Results

[0098] To verify the effectiveness of this invention compared to Kalantari19 and Chen21, two state-of-the-art HDR video reconstruction methods, this invention underwent extensive testing on several synthetic evaluation datasets. It was also compared with the few-sample HDR image generation method Prabhakar21 under the same conditions.

[0099] To evaluate performance, PSNR, HDR-VDP-2, and HDR-VQM were used as evaluation metrics. PSNR is the score between the generated result and the ground truth calculated using μ-law in the tone mapping domain. The calculation settings for HDR-VDP-2 and HDR-VQM were consistent with those in Chen21. Furthermore, PSNR-σ represents the average standard deviation of PSNR across all evaluation sequences, serving as a robustness metric. A lower PSNR-σ score, coupled with higher scores for other metrics, indicates that the trained model is robust and functioning correctly.

[0100] All methods were evaluated in a 5-shot, 5-way scenario, i.e., N uThe average score is fixed at 5, and the average score is reported by summing the results from 5 experiments. However, both traditional methods (Kalantari19 and Chen21) operate under supervised learning. For fairness, the traditional methods are also retrained, with the same data augmentation strategy, batch size, and batch size per cycle as the supervised branch of this method. In the results below, those marked with R are the retrained versions, with all training settings identical to those of this method.

[0101] The table below shows the quantitative evaluation results for the synthetic evaluation dataset in the case of a small sample size, where red indicates the best result and blue indicates the second best result:

[0102]

[0103] As can be seen, the method of this invention achieved the best results in both main metrics on the synthetic evaluation dataset. Although the score of HDR-VDP-2 is not the highest, it still surpasses the score of the original traditional method. Since the information contained in three consecutive frames is limited, the final generated HDR frame needs to be aligned with the intermediate frame in texture. Therefore, it is necessary to generate aligned neighboring frames as supplementary information, thus requiring 5 LDR frames. Re-exposure is a preprocessing step for the optical flow estimation network, which uses an existing network. After inference through 4 convolutional layers, 5 frames are generated, and this step concludes the process. The encoder-decoder obtains the output result by calculating the weight of each pixel in all input frames. The HDR frame is reconstructed from the output result of the average weighted value of the input frames.

[0104] Figure 5 The qualitative comparison of LDR reference frames underexposure or overexposure is shown on two sequences of the same synthetic evaluation dataset.

[0105] Figure 6 A qualitative comparison of visual effects on the same synthetic evaluation dataset is presented.

[0106] Figure 7 A qualitative comparison of the visual effects on the Kalantari13 dataset is presented.

[0107] It is evident from both qualitative and quantitative comparisons that, under demanding exposure conditions, the method of this invention outperforms traditional methods in small sample sizes, correcting ghosting and detail errors, removing noise, and resulting in a more visually pleasing image. The results from the aforementioned datasets demonstrate that the method of this invention can reconstruct high-quality HDR videos under different datasets and lighting conditions.

Claims

1. A method for reconstructing HDR videos based on semi-supervised learning, characterized in that, 1) Overall framework based on semi-supervised learning; 2) Merge network based on attention and hybrid dilated convolution; 3) Loss function based on semi-supervised learning; The process is as follows: 1-1) Prepare a small amount of labeled sample data, namely HDR video with ground truth and its corresponding LDR input video, and unlabeled sample data, namely LDR video without ground truth, and perform CRF replacement on the sample data to ensure a uniform format; 1-2) Input the two types of samples into the optical flow estimation network respectively to obtain aligned LDR frames. 1-3) The aligned LDR frames are input into the merge network, and the feature sequence is obtained through the hybrid dilated convolution module. Then, the attention module is used to give more weight to the key regions to correct the feature sequence. 1-4) The corrected feature sequences will be concatenated and input into the encoder-decoder. The encoder-decoder will obtain the output result by calculating the weight of each pixel in all input frames. The final HDR frame will be reconstructed from the output result of the average weighted value of the input frames. 1-5) The processes performed on the two types of samples can be regarded as two branches, namely supervised branch and unsupervised branch. Semi-supervised learning is performed by calculating and comparing the training loss functions under the two branches. Within the overall framework of semi-supervised learning, the semi-supervised learning framework utilizes LDR videos covering different real-world scenes to reconstruct HDR videos. With N l Each labeled sample is an HDR video with ground truth and its corresponding LDR video input, N l Small sample dataset D (no more than 5) l With N u A dataset D of unlabeled LDR videos, i.e., videos without ground truth. u As input to the neural network, the overall framework of semi-supervised learning includes a supervised branch fs and an unsupervised branch fu, which process the two inputs respectively: labeled LDR video input and unlabeled LDR video input. During training, the two video input branches have the same weight throughout the neural network. l With D u The samples are simultaneously used as inputs into the two branches; after the HDR video is reconstructed in the two video input branches respectively, the supervised loss function and the unsupervised loss function are calculated and used to update f; Merge network based on attention and hybrid dilated convolution; Hybrid dilated convolution modules are used for feature extraction of LDR frames, with input I... i Given I x F x h Feature sequence, m is the porosity, h m (•) represents the corresponding dilated convolutional layer, Concat is the concatenation operation, h(•) is the convolutional layer that reduces the size of the concatenated features, and the extraction process is defined as follows: After the hybrid dilated convolution module, the feature sequence F x h The features are extracted and fed into the attention module; the attention module is used to correct the feature sequences other than the reference frame. The attention module contains only two convolutional layers, and each attention module requires two inputs, namely features from neighboring frames and a reference frame; After sigmoid activation, the values ​​of the attention map are restricted to the range of [0,1]. The corrected feature sequences are concatenated and used as input to the encoder-decoder. The encoder-decoder calculates the weight of each pixel in all input frames to obtain the output result. The HDR frame is reconstructed and generated from the output result of the average weighted value of the input frames. The semi-supervised learning loss function is: The supervised loss function is L S The unsupervised loss function is L U β is the unsupervised loss function L U The weights are determined, therefore the total loss function L is: L = L S +βL U L S and L U The weighted sum is used as the final loss function, and the weights are updated via backpropagation using this loss function each time the network samples for training; L S As a supervised loss function, L plays a strong constraining role. U As an unsupervised loss function, it can be seen as a supplement, indirectly increasing the number of samples participating in training; a) Supervised loss function Supervised loss function L S From the absolute value loss function L m With perceptual loss function L p Composition, α is L p Weight: L S =L m +αL p To ensure that HDR frames generated by supervised branches are Approaching HDR Ground Truth An absolute value loss function was introduced. , and for and A tone-mapped image, defined as: Introducing a perceptual loss function pre-trained in the VGG-16 network , and For VGG-16 The features extracted by the layer are defined as follows: ; b) Unsupervised loss function Unsupervised loss function L U From the total variation loss function L t Consistency loss function L C Alignment loss function L a Composition, where θ and λ are the corresponding weights: L U =L t +θL C +λL a Total variational loss function L t It can improve the spatial smoothness of unsupervised branch generation of HDR frames, preserve the overall structure and details, and remove noise. and These are the horizontal and vertical difference operation matrices, respectively. The video output for the unsupervised branch is defined as follows: , Since the HDR frames generated by the network have a similar structure to the reference LDR frames, a consistency loss function is introduced. , Input video for the unsupervised branch of LDR. For the corresponding exposure level, The computational difference for SSIM is a sensory quantification of image quality. After adjusting the exposure, use The gamma curve transfers the HDR domain frame to the LDR domain; clip is a function that keeps the output value in [0,1]; the consistency loss function provides weak supervision to restore the correct shape and details of the reference frame, improving the visual experience of the generated HDR frame; it is defined as: , An alignment loss function L was introduced. a As a constraint to improve the results of the optical flow estimation network, for two alternately exposed LDR frames, the adjacent frames of the LDR reference frame have the same exposure; therefore, the alignment loss function is defined as the shape difference between the two. and Aligned LDR frames and Grayscale image: , The steps to reconstruct HDR video from a small number of labeled samples (HDR video) and their corresponding LDR input video, along with unlabeled samples (LDR video), are as follows: Preprocess the sample data and use CRF (Camera Response Function) replacement to unify the format of LDR input videos captured by different devices; L i For the correct LDR input frame after being replaced by CRF, F -1 For CRF, γ is a parameter of the gamma curve, set to 2.

2. The CRF replacement process is defined as follows: , The LDR input frame is fed into the optical flow estimation network, which is an improved version of SPyNet. It includes four consecutive convolutional layers. SPyNet is a well-performing deep learning-based optical flow estimation method that can achieve good results even when using video frames with different exposure levels as input. Therefore, this network is used as an auxiliary tool for optical flow estimation. Re-exposure is used to improve the performance of optical flow estimation. Inference calculations will then be performed in four consecutive convolutional layers of the optical flow estimation network. Reference frame L i It will be based on adjacent exposure t i+1 For reexposure, Clip is a function that keeps the output value within the range of [0, 1], γ is a parameter of the gamma curve, set to 2.2, and the reexposure process is defined as follows: , After reexposure and inference, the optical flow estimation map generated by the optical flow estimation network is used to transform adjacent frames to obtain five aligned LDR frames. Since the information contained in three consecutive frames is limited, the final generated HDR frame must be aligned with the intermediate frame in texture. Therefore, it is necessary to generate aligned neighboring frames as supplementary information, hence five LDR frames are required. These five frames are concatenated and mapped to the corresponding HDR domain, and the corresponding LDR input frame is generated by gamma correction along the channel dimension. ; The LDR input frames generated by the optical flow estimation network are fed into the merging network; after feature extraction by the hybrid dilated convolution module, the feature sequences are fed into the attention module for correction. The corrected feature sequences are concatenated and used as input to the encoder-decoder. The encoder-decoder calculates the weight of each pixel in all input frames to obtain the output result. The HDR frame is reconstructed from the output result of the average weighted value of the input frames.

Images