A Few-Shot HDR Ghost Removal Method Based on Saturation-Aware Encoder

Abstract

The present invention discloses a small-sample HDR ghost removal method based on a saturation-aware encoder. First, the short-exposure frame is adjusted to a new medium-exposure frame and a new long-exposure frame through exposure adjustment; then the LDR input image is mapped to the HDR domain through gamma correction; and then through the illusion module and the multi-scale residual Swin Transformer module, and through the convolutional layer, the final predicted image is obtained. The present invention only requires a small amount of labeled HDR data, meets the actual requirements, has strong applicability, and the reconstructed HDR image has high quality.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a small-sample HDR ghost removal method. Background Art

[0002] The development of high dynamic range (HDR) imaging is to solve the limitation that low dynamic range (LDR) images contain overexposed or underexposed areas, resulting in the destruction of scene details, and aims to generate HDR images by fusing multiple LDR images with different characteristics. However, in dynamic scenes or handheld camera scenes, ghosting phenomena will occur in HDR imaging. Common solutions include alignment-based methods, patch-based methods, and deep learning-based methods. However, these methods are prone to errors and require a large amount of labeled HDR data. The labeled HDR data requires an absolutely static background and a large amount of manpower for manual post-checking, making it difficult to collect. The literature "Multi-exposure Image Fusion Ghost Removal Algorithm Based on Patch Decomposition, Acta Optica Sinica, 2019, Vol 39(09), pp 132-140" discloses a multi-exposure image fusion ghost removal method based on patch decomposition. This method decomposes multi-exposure image patches into three conceptually independent parts: signal structure, signal intensity, and average intensity, uses the method of detecting the consistency of patch structures to detect ghosts, removes inconsistent patches and fuses these three parts separately, reconstructs the required patches and aggregates them into the final fused image. The method described in the literature is a patch decomposition-based method, still lacking in parameter adaptability and not very practical. In addition, this method requires a large amount of labeled HDR data for training, but due to the difficulty in collecting HDR data, it is difficult to obtain sufficient training data, restricting the practical application of this method. Summary of the Invention

[0003] In order to overcome the deficiencies of the prior art, the present invention provides a small-sample HDR ghost removal method based on a saturation-aware encoder. First, the short exposure frame is adjusted to a new medium exposure frame and a new long exposure frame through exposure adjustment; then the LDR input image is mapped to the HDR domain through gamma correction; and then through the illusion module and the multi-scale residual SwinTransformer module, and through the convolutional layer, the final predicted image is obtained. The present invention only requires a small amount of labeled HDR data, meets the actual needs, has strong applicability, and the reconstructed HDR image has high quality.

[0004] The technical solution adopted by the present invention to solve its technical problems includes the following steps:

[0005] Step 1: Adjust the short exposure frame through exposure adjustment to a new medium exposure frame and a new long exposure frame

[0006] Convert the short - exposure frame into a new medium - exposure frame and a long - exposure frame through exposure adjustment. The process is as follows: The process is as follows:

[0007]

[0008] where clip(·) represents exposure adjustment, γ represents the gamma - correction parameter, and t1, t2, and t3 represent the exposure times of the short - exposure frame, the medium - exposure frame, and the long - exposure frame, respectively;

[0009] Step 2: Map the LDR input image to the HDR domain through gamma correction to obtain H respectively to obtain H 1′ 、H 2′ 、H 3′ ;

[0010] The mapping process is as follows:

[0011]

[0012] where,

[0013] Step 3: Concatenate and H i′ to obtain the 6 - channel input I i , and mask the image patches of the input I i to obtain I′ i ;

[0014] Concatenate and H i′ along the channel dimension to obtain the 6 - channel input Subsequently, mask the image patches of the input I i to obtain I′ i ; specifically, divide the input into non - overlapping image patches and randomly mask a subset of these image patches at a high masking rate (75%) to obtain I′ i ;

[0015] Step 4: Input I′={I′1, I′2, I′3} into the hallucination module to obtain features

[0016] First, input I′1, I′2, and I′3 into three convolutional layers respectively to extract the shallow - layer features F i ; then, divide the shallow - layer features F i into non - overlapping image patches and for each image patch Map to query q, key k, and value v; subsequently, calculate the similarity between query q and key k, and use the Softmax function to obtain the attention weights; finally, apply the attention weights to v to obtain The operation process is as follows:

[0017]

[0018]

[0019] In the formula, W q , W k , W v respectively represent The parameter matrices for mapping to query q, key k, and value v, respectively represent the query, key, and value after mapping of the image patches, b is the learnable position encoding, and d is the dimension of;

[0020] Step 5: Concatenate along the channel dimension to obtain F s , and then input it into the multi-scale residual Swin Transformer module to output the feature F out ;

[0021] First, concatenate along the channel dimension to obtain the input of the multi-scale residual Swin Transformer module; the multi-scale residual Swin Transformer module consists of multiple multi-scale Swin Transformer layers STL, multiple convolutional layers, and a residual connection; given the input feature of the i-th multi-scale residual Swin Transformer module, the output of the i-th multi-scale residual Swin Transformer module is expressed as follows:

[0022]

[0023]

[0024] In the formula, represents the N-th Swin Transformer layer with scale l j in the i-th multi-scale residual Swin Transformer module; after passing through multiple multi-scale residual Swin Transformer modules, the final output is F out ;

[0025] Step 6: F out passes through a convolutional layer and the residual F obtained by concatenating along the channel in Step 5s Add them together to obtain the final predicted image

[0026] F out Input it into the convolutional layer and then add it to the residual F s Add them together to obtain the final predicted image The calculation process is as follows:

[0027]

[0028] Step 7: Design the loss function and train the model;

[0029] Use the function ω to transform the predicted HDR image into short-exposure, medium-exposure, and long-exposure LDR images The process is as follows:

[0030]

[0031] where i = 1, 2, 3 correspond to short-exposure, medium-exposure, and long-exposure respectively;

[0032] Convert the short-exposure frame into new short, medium, and long-exposure frames through label generation; then, use the new exposure frames as the labels of the model The calculation process is as follows:

[0033]

[0034] Finally, calculate and the L1 self-supervised loss between them:

[0035]

[0036] Step 8: Use the static labeled LDR samples S and the dynamic labeled LDR samples D to fine-tune the trained model to obtain an optimized model;

[0037] Use the dynamic labeled samples D and the static labeled samples S to fine-tune the model, and apply the μ-law mapping to map the linear domain image to the tone mapping domain image as follows:

[0038]

[0039] where T(x) is the tone mapping function;

[0040] Then input the dynamic labeled samples D and the static labeled samples S into the model to obtain the predicted HDR image and and calculate the reconstruction loss L between it and the sample label recon and the perceptual loss L percep, the formula is as follows:

[0041]

[0042]

[0043] L finetune = L recon + λL percep (14)

[0044] In the formula, σ i,j (.) represents the j-th convolutional layer and the i-th max pooling layer in the VGG19 network, and λ = 1e -2 ;

[0045] Step 9: Use the unlabeled LDR samples U to iteratively optimize the model;

[0046] Generate HDR pseudo-labels for the unlabeled data and use the unlabeled LDR samples U and the labeled LDR samples S, D to iteratively optimize the model;

[0047] Step 9-1: Adopt an adaptive pseudo-label selection strategy to select ghost-free HDR pseudo-labels;

[0048] At time step t, use the optimized model N t to predict the HDR images of the dynamic and static labeled samples and Then use the function ω to map the predicted HDR images to medium-exposure images and calculate the loss with the original medium-exposure LDR image The formula is as follows: The formula is as follows:

[0049]

[0050] In the formula, mask(·) represents covering the overexposed and underexposed areas; the area with the brightness value between τ0 and τ1 is set as the normally exposed area, less than τ0 is underexposed, and greater than τ1 is overexposed;

[0051] Sort the losses of all image patches and use the σ(·,·) function to obtain the β-percentile loss as the selection threshold τ t :

[0052]

[0053] Use the model N t to predict the pseudo-labels of the unlabeled samples Then use the ω function to map to medium exposure to obtain And calculate and the original medium-exposure LDR image to obtain the loss between them If the current loss is greater than the threshold τ t , it is determined that the quality of the pseudo-label is poor, with more saturated regions and ghost regions, and a weight is given, which tends to linearly decay in the next training iteration. The calculation process is as follows:

[0054]

[0055]

[0056]

[0057] In the formula, is the medium-exposure image without labels at time step t, is the maximum selection loss of the unlabeled samples at time step t, is the unlabeled sample U i the weight factor in the (t + 1)-th training iteration;

[0058] Step 9-2: Iteratively and adaptively optimize the model using the dynamic and static labeled samples D and S and the unlabeled sample U;

[0059] At time step t, use the model N t to predict the pseudo-labels of the unlabeled data Then train the model N with the labels and the pseudo-labeled samples t to obtain the model N at time step t + 1 t+1 ; In each time step, calculate the reconstruction loss and the perceptual loss as follows:

[0060]

[0061] In the formula, λ = 1e -2 , is the weight factor of the unlabeled data U i .

[0062] Preferably, the gamma correction parameter γ = 2.2.

[0063] Preferably, the μ = 5000, β = 85.

[0064] The beneficial effects of the present invention are as follows:

[0065] To overcome the deficiencies of existing multi-exposure image fusion deghosting algorithms based on image block decomposition, the present invention provides a small-sample HDR deghosting method based on a saturation-aware encoder. Considering that the saturated regions can be regarded as masking the low-dynamic-range (LDR) input regions, the present invention proposes a self-supervised learning network that uses unlabeled LDR data to learn robust feature representations and reconstruct non-saturated HDR images. To solve the ghosting problem, the present invention proposes an iterative semi-supervised learning method based on sample quality, which uses a small number of labeled HDR samples to fine-tune the model. To prevent overfitting problems and make full use of unlabeled data, the present invention also proposes an adaptive pseudo-label selection strategy to select high-quality HDR pseudo-labels to avoid the influence of mislabeled samples. This method first solves the saturation problem and then solves the deghosting problem. It adopts self-supervised and semi-supervised learning methods and only requires a small number of labeled HDR data, meeting the actual needs, having strong applicability, and high-quality reconstructed HDR images. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 FIG. is a flowchart of deghosting of the HDR image of the present invention.

[0067] Figure 2 FIG. is a short-exposure frame of an unlabeled LDR image according to an embodiment of the present invention.

[0068] Figure 3 FIG. is a new medium-exposure frame after exposure adjustment according to an embodiment of the present invention.

[0069] Figure 4 FIG. is a new medium-exposure frame after exposure adjustment according to an embodiment of the present invention.

[0070] Figure 5 FIG. is a predicted HDR image according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0071] The present invention will be further described below in conjunction with the drawings and embodiments.

[0072] The following takes the short-exposure frame of the unlabeled LDR image as an example, as Figure 2 shown, to describe the specific implementation manner. The small-sample HDR deghosting method based on the saturation-aware encoder in this embodiment includes the following steps, as Figure 1 shown:

[0073] Step 1: Adjust the short-exposure frame to a new medium-exposure frame and a new long-exposure frame as Figure 3 , Figure 4 shown.

[0074] Because the medium-exposure frame of the unlabeled data U and long exposure frames There are more saturated regions in the short exposure frames. First, the present invention converts the short exposure frames into new medium exposure frames and long exposure frames through exposure adjustment. The process is as follows: and long exposure frames The process is as follows:

[0075]

[0076] In the formula, clip(·) represents exposure adjustment, γ represents the gamma correction parameter, and t1, t2, and t3 respectively represent the exposure time of the short exposure frame, the exposure time of the medium exposure frame, and the exposure time of the long exposure frame.

[0077] Step 2: Map the LDR input image to the HDR domain through gamma correction to obtain H 1′ 、H 2′ 、H 3′ .

[0078] Since gamma correction basically maps the input image to a domain closer to human perception, the present invention uses it as a mapping function. The mapping process is as follows:

[0079]

[0080] In the formula, t i represents the exposure time of the LDR image , and γ is the gamma correction parameter. The present invention sets γ to 2.2.

[0081] Step 3: Connect and H i′ to obtain the 6-channel input I i , and mask the image blocks of the input I i to obtain I′ i , i = 1, 2, 3.

[0082] Connect and H i′ along the channel dimension to obtain the 6-channel input Subsequently, mask the image blocks of the input I i to obtain I′ i . Specifically, the input is divided into non-overlapping image blocks, and a subset of these image blocks is randomly masked at a high masking rate (75%). Considering that the masking strategy is another method to destroy saturated regions, the present invention enables the model to learn a robust representation to recover these saturated regions, and I′ = {I′1, I′2, I′3} is the input of the model.

[0083] Step 4: Input I′ into the illusion module to obtain features i = 1, 2, 3.

[0084] First, input I′ into three convolutional layers to extract shallow features F i . Then, divide the shallow features F i into non - overlapping image patches and map each image patch to query q, key k, and value v. Subsequently, calculate the similarity between q and k, and use the Softmax function to obtain the attention weights. Finally, apply the attention weights to v to obtain The operation process is as follows:

[0085]

[0086]

[0087] where W q , W k , W v represent the parameter matrices for mapping to query q, key k, and value v respectively, b is the learnable positional encoding, and d is the dimension of .

[0088] Step Five: Concatenate along the channel dimension to obtain F s , and then input it into the multi - scale residual SwinTransformer module to output the feature F out .

[0089] First, concatenate along the channel dimension to obtain the input of the multi - scale residual Swin Transformer module. Then, the multi - scale residual Swin Transformer module combines a large amount of information from different exposure regions. The multi - scale residual Swin Transformer module consists of multiple multi - scale Swin Transformer layers (STL), several convolutional layers, and a residual connection. Given the input feature of the i - th multi - scale residual Swin Transformer module, the output of the multi - scale residual Swin Transformer module can be expressed as follows:

[0090]

[0091]

[0092] where represents the scale l in the i - th multi - scale residual Swin Transformer modulej The Nth Swin Transformer layer, Represents the input features of the Nth SwinTransformer layer in the i-th multi-scale residual Swin Transformer module. After passing through multiple multi-scale residual Swin Transformer modules, the final output F out .

[0093] Step 6: F out After the convolution layer and the residual F s Add together to get the final predicted image like Figure 5 shown.

[0094] F out The input convolution layer is then combined with the residual F s Add together to get the final predicted image The calculation process is as follows:

[0095]

[0096] Step 7: Design the loss function and train the above model.

[0097] Since we use unlabeled data U, there is no real label for the HDR image. We calculate the self-supervised loss in the LDR domain. First, we use the function ω to transform the predicted HDR image Transform to short exposure, medium exposure and long exposure LDR images The process is as follows:

[0098]

[0099] Where i=1, 2, 3 represents the exposure level of the LDR image, t i is the exposure time, and γ is the gamma correction parameter. In order to restore the saturated area, the present invention converts the short exposure frame into new short, medium and long exposure frames through label generation. Then, the new exposure frame is used as the label of the model The calculation process is as follows:

[0100]

[0101] Final calculation and The L1 self-supervised loss between:

[0102]

[0103] Step 8: Fine-tune the trained model using static labeled LDR samples S and dynamic labeled LDR samples D.

[0104] To improve the saturation region and further learn to process the ghosting region, the present invention fine-tunes the pre-trained model using some dynamic labeled samples D and static labeled samples S. Here, the "μ-law" is applied to map the linear domain image to the tone mapping domain image as follows:

[0105]

[0106] where T(x) is the tone mapping function and μ = 5000. Then, the two types of samples are input into the model to obtain the predicted HDR image and and calculate the reconstruction loss L between it and the sample label recon and the perceptual loss L percep , and the formulas are as follows:

[0107]

[0108]

[0109]

[0110] where σ i,j (·) represents the j-th convolutional layer and the i-th max pooling layer in the VGG19 network, and λ = 1e -2 . The model is fine-tuned through the final fine-tuning loss L finetune .

[0111] Step Nine: Use the unlabeled LDR samples U to iteratively optimize the model.

[0112] To prevent the overfitting problem of a small number of labeled training samples and utilize the unlabeled samples, the present invention further generates HDR pseudo-labels for the unlabeled data in Step Nine and iteratively optimizes the model using the unlabeled LDR samples U and the labeled LDR samples S, D.

[0113] Sub-step 1: Adopt an adaptive pseudo-label selection strategy to select well-exposed and ghost-free HDR pseudo-labels.

[0114] Since the HDR pseudo-labels inevitably contain saturated and ghosting samples, the present invention proposes an adaptive pseudo-label selection strategy to select well-exposed and ghost-free HDR pseudo-labels to avoid the pseudo-labels hindering the optimization process. Specifically, at time step t, the model N t is used to predict the HDR images of the dynamic and static labeled samples and Then, the function ω is used to map the predicted HDR image to a medium-exposure image and calculate with the original medium-exposure LDR image Loss in well-exposed regions The formula is as follows:

[0115]

[0116] Where mask(·) represents covering overexposed and underexposed regions. Subsequently, the losses of all image patches are sorted, and the σ(·,·) function is used to obtain the β-percentile loss as the selection threshold τ t :

[0117]

[0118] Where β is set to 85. In addition, model N t is used to predict the pseudo-labels of unlabeled samples Similar to the operation of the labeled data above. Then the ω function is used to map to medium exposure to obtain and calculate between and the original medium-exposure LDR image to obtain the loss If the current loss is greater than the threshold τ t , it is considered that the pseudo-label quality is poor, with more saturated regions and ghost regions, then a lower weight will be given, and this weight tends to linearly decay in the next training iteration. The calculation process is as follows:

[0119]

[0120]

[0121]

[0122] Where is the medium-exposure image of the unlabeled at time step t, is the maximum selection loss of the unlabeled samples at time step t, is the weight factor of the unlabeled sample U i at the (t + 1)-th training iteration.

[0123] Sub-step 2: Iteratively and adaptively optimize the model using a small number of dynamic and static labeled samples D and S and a large number of unlabeled samples U.

[0124] Specifically, at time step t, model N t is used to predict the pseudo-labels of unlabeled data Then model N t is trained with some labeled and pseudo-labeled samples to obtain model N at time step t + 1 t+1In each time step, the reconstruction loss and perceptual loss are calculated as follows:

[0125]

[0126] where λ = 1e -2 , is the weight factor of the unlabeled data U i .

Claims

1. A small-sample HDR ghost removal method based on a saturation-aware encoder, characterized in that, It includes the following steps: Step 1: Adjust the short exposure frame to a new medium exposure frame and a new long exposure frame through exposure adjustment Convert the short exposure frame into a new medium exposure frame and a long exposure frame through exposure adjustment. The process is as follows: ​ In the formula, clip(·) represents exposure adjustment, γ represents the gamma correction parameter, and t1, t2, and t3 respectively represent the exposure time of the short-exposure frame, the exposure time of the medium-exposure frame, and the exposure time of the long-exposure frame; Step 2: Map the LDR input image to the HDR domain through gamma correction to obtain H , H 1′ , H 2′ , and H 3′ ; The mapping process is as follows: In the formula, Step 3: Connect and H i′ to obtain a 6-channel input I i , and mask the input I i image blocks to obtain I' i ; Concatenate along the channel dimension and H i′ to obtain a 6-channel input Subsequently, mask the image patches of input I i to obtain I′ i Specifically: divide the input into non-overlapping image patches and randomly mask a subset of these image patches with a high masking rate (75%) to obtain I′ i ; Step 4: Input I′ = {I′1, I′2, I′3} into the illusion module to obtain features First, input I′1, I′2, and I′3 into three convolutional layers respectively to extract shallow features F i ; Then, divide the shallow features F i into non-overlapping image patches and map each image patch to query q, key k, and value v; Subsequently, the similarity between the query q and the key k is calculated, and the Softmax function is used to obtain the attention weights; finally, the attention weights are applied to v to obtain The operation process is as follows: Where, W q , W k , W v respectively represent the parameter matrices mapped to the query q, the key k, and the value v, respectively represent the query, key, and value after the mapping of the image patch, b is the learnable positional encoding, and d is the dimension of Step 5: Connect along the channel to obtain F s , then input it into the multi-scale residual Swin Transformer module, and output the feature F out ; First, are concatenated along the channel dimension to obtain the input of the multi-scale residual Swin Transformer module; the multi-scale residual Swin Transformer module consists of multiple multi-scale Swin Transformer layers STL, multiple convolutional layers, and a residual connection; given the input feature of the i-th multi-scale residual Swin Transformer module, the output of the i-th multi-scale residual Swin Transformer module is expressed as follows: wherein, represents the Nth Swin Transformer layer with scale l in the ith multi-scale residual Swin Transformer module; after passing through multiple multi-scale residual Swin Transformer modules, the final output is F j ; out ; Step 6: F out After passing through the convolutional layer and adding to the residual Fs obtained by channel connection in Step 5, the final predicted image is obtained F out Input the convolutional layer and then add it to the residual F s Add them together to obtain the final predicted image The calculation process is as follows: Step 7: Design a loss function and train the model; Use the function ω to transform the predicted HDR image into short-exposure, medium-exposure, and long-exposure LDR images The process is as follows: In the formula, i = 1, 2, 3 respectively correspond to short exposure, medium exposure, and long exposure; Convert the short-exposure frame into new short, medium, and long-exposure frames through label generation; Then, use the new exposure frame as the label of the model The calculation process is as follows: Final calculation and The L1 self-supervised loss between: Step 8: Use the static labeled LDR samples S and the dynamic labeled LDR samples D to fine-tune the trained model to obtain an optimized model; Use the dynamic labeled samples D and the static labeled samples S to fine-tune the model, and apply the μ-law mapping to map the linear domain image to the tone mapping domain image, as follows: In the formula, T(x) is the tone mapping function; Then, input the dynamic marker sample D and the static marker sample S into the model to obtain the predicted HDR image and calculate its reconstruction loss L with the sample label recon and perceptual loss L percep , as shown in the following formula: L finetune = L recon + λL percep (14) where, σ i，j (·) represents the j-th convolutional layer and the i-th max pooling layer in the VGG19 network, and λ = 1e -2 ; Step 9: Use the unlabeled LDR samples U to iteratively optimize the model; Generating HDR Pseudo-Labels for Unlabeled Data And iteratively optimize the model using unlabeled LDR samples U and labeled LDR samples S, D; Step 9-1: Adopt an adaptive pseudo-label selection strategy to select ghost-free HDR pseudo-labels; At time step t, the optimization model N is used t to predict the HDR images of dynamic and static marker samples and Then, the predicted HDR images are mapped to medium-exposure images using the function ω and the loss with the original medium-exposure LDR image is calculated as follows: In the formula, mask(·) represents covering the overexposed and underexposed areas; set the brightness value between τ0 and τ1 as the normally exposed area, less than τ0 as underexposed, and greater than τ1 as overexposed; Sort the losses of all image patches and use the σ(·, ·) function to obtain the β-percentile loss as the selection threshold τ t : Using model N t Predict the pseudo - labels of unlabeled samples Then use the ω function to Map to medium exposure to obtain And calculate And the original medium - exposure LDR image The loss between them to obtain If the current loss Is greater than the threshold τ t , it is determined that the pseudo - label quality is poor, with more saturated regions and ghost regions, then a weight is given, which tends to linearly decay in the next training iteration. The calculation process is as follows: wherein, is the medium-exposure image without labels at time step t, is the maximum selection loss of the unlabeled samples at time step t, is the unlabeled sample U i is the weight factor in the (t + 1)-th training iteration; Step 9-2: Use the dynamic and static labeled samples D and S and the unlabeled samples U to iteratively and adaptively optimize the model; At time step t, model N is used t to predict the pseudo-labels of unlabeled data Then, model N is trained with the labeled and pseudo-labeled samples t to obtain model N at time step t+1 t+1 ; In each time step, the reconstruction loss and perceptual loss are calculated as follows: where λ = 1e -2 , is the weight factor of the unlabeled data U i .

2. The small-sample HDR ghost removal method based on a saturation-aware encoder according to claim 1, wherein The gamma correction parameter γ = 2.

2.

3. A small-sample HDR ghost removal method based on a saturation-aware encoder according to claim 1, characterized in that, The μ = 5000 and β = 85.

Images