Defocusing and deblurring method based on hierarchical model

Abstract

The invention discloses a defocusing and deblurring method based on a hierarchical model, and relates to a defocusing and deblurring method. The invention aims to solve the problem of incompatibility between a high-level visual demand and an image recovery task. The method comprises the following steps of: 1, segmenting a blurred image x belonging to RC * H * W into h belonging to RN * D by using a basic feature encoder; step 2, a context encoder uses # imgabs0 # to learn a clearer and abstract representation hs; and step 3, using the # imgabs1 # and the hs as input by the decoder, and reconstructing an image # imgabs2 # after the blurring is removed. The invention belongs to the technical field of image processing.

Description

Technical Field

[0001] The invention relates to a defocus deblurring method and belongs to the technical field of image processing. Background Art

[0002] Defocus deblurring has always been a challenging problem in dealing with spatial blur caused by scene depth variations. Although recent work has made many advances in network architecture design, these advances mainly focus on the processing of high-frequency details. The importance of scene understanding in the deblurring process has not been given enough attention. The core of scene understanding lies in the use of contextual information, which provides key support for capturing high-level semantic cues of the environment and object contours. By identifying and effectively utilizing these cues, the quality of image restoration can be significantly improved. Based on this, we propose a new method that combines spatial details and contextual information to provide an effective solution to the defocus deblurring problem. Specifically, we introduce a novel hierarchical model based on the Visual Transformer (ViT), which can seamlessly integrate spatial details and contextual information. Our method decomposes the deblurring task into two independent subtasks: the first subtask is handled by the base feature encoder, which converts the blurred image into a specific and detailed base feature representation; the second subtask is handled by the context encoder, which extracts abstract and clear representations from the base feature representation. The outputs of the two encoders are combined and reconstructed into a clear target image through a decoder.

[0003] Defocus blur occurs when the depth range of a scene exceeds the depth of field (DoF) limit of a given camera. In this case, objects located in out-of-focus areas will appear blurred, resulting in the loss of key details. This image degradation has a significant negative impact on a variety of computer vision applications, including object detection, image super-resolution, and text recognition. Therefore, restoring an all-focus image to reveal its high-resolution details has important research significance and practical application value in the field of computer vision. To deal with the defocus blur problem, researchers have developed a variety of algorithms that can transform blurry images into sharp images. These methods usually involve the use of specific network modules, such as dynamic kernels and multi-scale modules, which have shown promising performance in standard defocus deblurring benchmarks. The current mainstream research trend is to combine advanced network architectures with accurately modeled defocus priors to achieve the best deblurring effect. Representative models such as NRKNet and KPAC reflect the actual results of this trend. The above deblurring algorithms mainly focus on reconstructing the high-frequency detail features of the image, but often fail to fully capture the so-called "contextual information", that is, high-level semantic clues. This type of information is closely related to the semantics or context of the input image and is particularly important for defocus deblurring, where the deblurring process needs to be performed separately for different regions of the image. At present, deblurring networks try to obtain a wide receptive field by using large-scale kernels and self-attention mechanisms. However, these networks often only focus on capturing local spatial related information and fail to deeply understand the overall semantics of the scene. In the field of computer vision, in-depth exploration of image semantic context information has made significant progress recently. This trend coincides with the demand for semantic information in defocus deblurring, providing new possibilities for breaking through the limitations of existing methods. In this area, the Vision Transformer (ViT) architecture is particularly prominent. Models built on ViT, such as MAE and CLIP, have demonstrated their excellent image understanding capabilities even when dealing with degraded inputs. These advances highlight the great potential of incorporating high-level semantic context information into the defocus deblurring task. However, the incompatibility between high-level visual requirements and image restoration tasks remains a significant challenge. Summary of the invention

[0004] In order to solve the incompatibility problem between high-level visual requirements and image restoration tasks, the present invention further proposes a defocus deblurring method based on a hierarchical model.

[0005] The technical solution adopted by the present invention to solve the above problems is: the present invention specifically includes:

[0006] Step 1: Use the basic feature encoder to transform the blurred image x∈R C×H×W Split into h∈R N×D ;

[0007] Step 2: Context Encoder Use Learn clearer and more abstract representations h s ;

[0008] Step 3: The decoder will and h s As input, reconstruct the deblurred image

[0009] Furthermore, in step 2, the context encoder uses Learn clearer and more abstract representations h s Specifically include:

[0010] Using a linear layer from Prediction s :

[0011]

[0012] In formula (1), W represents the weight of the linear layer;

[0013] Reference Features Obtained by applying the same encoder to the reference image y, the dimension of the reference image y is R C×H×W ;

[0014] The introduction of the classification loss function based on the second-level cross entropy promotes and Learn features that distinguish blurry from sharp images:

[0015]

[0016]

[0017] In formula (2), t represents the label, and t∈{0,1}, 0 represents blur, 1 represents clear, and N represents the tensor h * The total number of elements in , σ represents the sigmoid function, which is used to normalize the eigenvalue interval, and the eigenvalue interval is (0,1);

[0018] The overall loss of the blur-to-sharp converter module is defined as:

[0019]

[0020] In formula (3), d(·) is used to calculate the distance between representations, and the hyperparameter λ is used to balance the importance of classification loss and other loss components;

[0021] By using the loss function l BSTThe context encoder is trained to obtain the representation related to ambiguity and predict the s Clear representation of representation.

[0022] Furthermore, a joint embedding architecture is introduced in the context encoder to enhance the representation h s The architecture includes an information content preservation module, which uses the regularization standard defined by VICReg to remove details irrelevant to the blur. The information content preservation module maps the representation to an embedding space through a multi-layer perceptron and derives a regularization loss from it.

[0023] Regularization losses are used in the information content preservation module. These losses are based on three principles: variance, invariance, and covariance. When calculating the regularization loss, the information content preservation module receives two sets of inputs: a batch of reference representations and the representation h converted from the fuzzy-sharp converter s , through MLPf θ (·) Embed the above representation into a low-dimensional space to enhance the abstraction capability:

[0024]

[0025] In formula (4), e(·) represents the feature average operator, The dimension is R D , excluding the [CLS]| token used for classification in the transformer architecture when calculating the average; two sets of batch embeddings are derived from the mapping, denoted as and Where m is the batch size;

[0026] The regularized loss function of the information content preserving module is defined as:

[0027]

[0028] In formula (5), z represents or γ and ε represent hyperparameters, represents the mean of the batch embedding, i.e. The total loss function of the information content preservation module contains three regularization terms, positioned as:

[0029]

[0030] In formula (6), μ1 and μ2 are hyperparameters used to balance the impact of various loss terms.

[0031] Furthermore, the decoder uses the basic feature encoder and the context encoder to extract and h s Generate deblurred image The decoder is trained using the traditional l1 loss function:

[0032]

[0033] For the entire model, the final loss is defined as:

[0034]

[0035] In formula (8), α and β are parameters that balance different loss terms.

[0036] Furthermore, the training strategy for training the decoder is as follows: the base feature encoder parameterized by the pre-trained MAE is frozen during the training of the context encoder, and then this model is trained on the large-scale ImageNet dataset; widely accepted data augmentation techniques are adopted, including random cropping to a size of 224×224, random scaling from 0.2 to 1.0, and random horizontal flipping; the loss of the entire model is minimized according to the loss function defined in formula (8), and the context encoder is initialized using the parameters of the pre-training stage.

[0037] The beneficial effects of the present invention are: on the metrics directly related to human visual perception, such as MUSIQ, the present invention is 1.65 points higher than the existing state-of-the-art NRKNet; compared with those methods designed with specific auxiliary input structures, such as KPAC, the present invention also achieves a significant performance improvement of 0.042 on FSIM; the indicators of the present invention better reflect human visual perception, and the present invention has significant potential in the single image defocus deblurring task;

[0038] The present invention effectively restores the content structure of the image, while other methods usually leave significant background blur. The present invention successfully reconstructs the image structure, makes the edges sharper, and produces a clear and visually pleasing image. The present invention can more accurately restore details, reduce artifacts, and accurately handle defocused areas.

[0039] The present invention integrates high-level context information into the single image defocus deblurring task. It makes full use of the ability of pre-trained ViT in extracting high-level context representations; the present invention effectively handles the extraction of abstract features for semantic understanding and the retention of high-frequency details, and effectively alleviates the potential conflict between high-level visual tasks and image restoration tasks; the present invention decomposes the deblurring task into two independent subtasks, and processes complex blurry scenes through two specialized encoders, thereby effectively eliminating the defocus blur caused by spatial changes;

[0040] The present invention integrates high-level semantic context information and high-frequency details into the defocus deblurring task, achieving an effective balance between these two key aspects; it innovatively decomposes the image deblurring task into two parts: abstract context feature extraction and specific detail extraction; this effectively makes up for the shortcomings of existing defocus deblurring processing methods in context feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the layered model;

[0042] Figure 2 It is a schematic diagram of visual comparison of single image defocus deblurring performed on the DPDD dataset of the present invention. DETAILED DESCRIPTION

[0043] Specific implementation method 1: Figure 1 and Figure 2 As shown, a defocus deblurring method based on a hierarchical model specifically includes:

[0044] Step 1: Use the basic feature encoder to transform the blurred image x∈R C×H×W Split into h∈R N×D ;

[0045] Step 2: Context Encoder Use Learn clearer and more abstract representations h s ;

[0046] Step 3: The decoder will and h s As input, reconstruct the deblurred image

[0047] Among them, the defocus deblurring method based on the hierarchical model is based on the standard ViT architecture, which includes a basic feature encoder, a context encoder and a decoder. The basic feature encoder accepts the segmented h as input and is committed to learning a primary representation that retains as many details as possible. This representation not only preserves the integrity of the input, but also distinguishes different levels of blur. The context encoder then uses Learn clearer and more abstract representations h s , effectively eliminating irrelevant fuzzy features. The decoder then converts and h s As input, reconstruct the deblurred image

[0048] The goal of the basic feature encoder is to map the blurred image into the latent space and extract the primary representation Aims to generate restored high-fidelity input to assist the context encoder; uses MAE as the framework of the basic feature encoder to extract primary representations The architecture of MAE is the same as the standard ViT; I use several early layers of MAE as the basic feature encoder to keep the model simple and feasible for optimization, and initialize it with parameters trained on MIM.

[0049] By using the loss function l BST The context encoder is trained to obtain the representation related to ambiguity and predict the s Clear representation of representation.

[0050] Specific implementation method 2: Figure 1 and Figure 2 As shown, the context encoder in step 2 uses Learn clearer and more abstract representations h s Specifically include:

[0051] Using a linear layer from Prediction s :

[0052]

[0053] In formula (1), W represents the weight of the linear layer;

[0054] Reference Features Obtained by applying the same encoder to the reference image y, the dimension of the reference image y is R C×H×W ;

[0055] The introduction of the classification loss function based on the second-level cross entropy promotes and Learn features that distinguish blurry from sharp images:

[0056]

[0057]

[0058] In formula (2), t represents the label, and t∈{0,1}, 0 represents blur, 1 represents clear, and N represents the tensor h * The total number of elements in , σ represents the sigmoid function, which is used to normalize the eigenvalue interval, and the eigenvalue interval is (0,1);

[0059] The overall loss of the blur-to-sharp converter module is defined as:

[0060]

[0061] In formula (3), d(·) is used to calculate the distance between representations, and the hyperparameter λ is used to balance the importance of classification loss and other loss components;

[0062] By using the loss function l BST The context encoder is trained to obtain the representation related to ambiguity and predict the s Clear representation of representation.

[0063] Specific implementation method three: Figure 1 and Figure 2 As shown in Figure 2, a joint embedding architecture is introduced in the context encoder to enhance the representation h s The architecture includes an information content preservation module, which uses the regularization standard defined by VICReg to remove details irrelevant to the blur. The information content preservation module maps the representation to an embedding space through a multi-layer perceptron and derives a regularization loss from it.

[0064] Regularization losses are used in the information content preservation module. These losses are based on three principles: variance, invariance, and covariance. When calculating the regularization loss, the information content preservation module receives two sets of inputs: a batch of reference representations and the representation h converted from the fuzzy-sharp converter s , through MLPf θ (·) Embed the above representation into a low-dimensional space to enhance the abstraction capability:

[0065]

[0066] In formula (4), e(·) represents the feature average operator, The dimension is R D , excluding the [CLS]| token used for classification in the transformer architecture when calculating the average; two sets of batch embeddings are derived from the mapping, denoted as and Where m is the batch size;

[0067] The regularized loss function of the information content preserving module is defined as:

[0068]

[0069] In formula (5), z represents or γ and ε represent hyperparameters, represents the mean of the batch embedding, i.e. The total loss function of the information content preservation module contains three regularization terms, positioned as:

[0070]

[0071] In formula (6), μ1 and μ2 are hyperparameters used to balance the impact of various loss terms.

[0072] Specific implementation method four: Figure 1 and Figure 2 As shown, the decoder uses the basic feature encoder and the context encoder to extract and h s Generate deblurred image The decoder is trained using the traditional l1 loss function:

[0073]

[0074] For the entire model, the final loss is defined as:

[0075]

[0076] In formula (8), α and β are parameters that balance different loss terms.

[0077] Specific implementation method five: Figure 1 and Figure 2 As shown in Figure 1, the training strategy for training the decoder is as follows: the basic feature encoder parameterized by the pre-trained MAE is frozen during the training of the context encoder, and then this model is trained on the large-scale ImageNet dataset; widely accepted data augmentation techniques are used, including random cropping to a size of 224×224, random scaling from 0.2 to 1.0, and random horizontal flipping; the loss of the entire model is minimized according to the loss function defined in formula (8), and the context encoder is initialized using the parameters of the pre-training stage.

[0078] Dataset and implementation details

[0079] The DPDD dataset was collected using the dual-pixel technology of the Canon EOS 5D Mark IV digital SLR camera. It contains 500 different indoor and outdoor scenes. For each scene, two defocused sub-views were captured using a wide aperture, and the corresponding clear real image was captured using a narrow aperture. The dataset is divided into 350 pairs of training samples, 74 pairs of verification samples, and 76 pairs of test samples.

[0080] Implementation details: The basic feature encoder and decoder each contain four ViT layers, the context encoder consists of three ViT layers, and the information content preservation (ICR) module consists of three fully connected layers with batch normalization. According to the configuration of VICReg, l BST The coefficient λ in the loss is set to 0.25, and l ICR The μ1 and μ2 in the loss are set to 0.25 and 0.01 respectively, and the loss function of the overall model is l ALLIn the experiment, the hyperparameters α and β are set to 1. The experiment is conducted using the PyTorch framework on a platform equipped with an RTX 3090 GPU. In the pre-training phase on the ImageNet dataset, the LARS optimizer is used for training for 100 cycles. In the training phase of the defocus dataset, the AdamW optimizer is used, and its β1 and β2 are 0.9 and 0.99, respectively. The initial learning rate is set to 2e-4 and gradually decreases to 2e-5 within 200 cycles. The batch size is set to 8. When using ViT-B as the backbone model, the same word segmentation method as the original ViT-B is used, and the block size is 16×16. In the test phase, in order to process high-resolution blurred images, the images are segmented into 320×320 blocks, and the sliding window method is used to optimize computational efficiency.

[0081] Defocus dataset effect evaluation

[0082] Six standard evaluation metrics were selected, including peak signal-to-noise ratio (PSNR), structural similarity (SSIM), learned perceptual patch similarity (LPIPS), multi-scale image quality estimator (MUSIQ), feature similarity (FSIM) and conditional knowledge distillation network (CKDN); among them, LPIPS, MUSIQ and FSIM are closely related to human visual perception.

[0083] Table 1: Defocus deblurring performance comparison; This table summarizes the defocus deblurring comparisons on the DPDD test dataset, RealDOF dataset, and LFDOF dataset. The best and second-best results in each metric are marked in bold and underlined, respectively, to facilitate quick identification of model performance.

[0084] Table 1

[0085]

[0086] Table 1 shows the quantitative comparison results of the present invention with all current mainstream methods (SOTA) on the DPDD dataset in detail. Although the present invention adopts the ViT-based architecture commonly used to process high-level visual tasks, which is not specially designed for image restoration, it shows excellent performance in various evaluation indicators. In particular, in metrics directly related to human visual perception, such as MUSIQ, the present invention is 1.65 points higher than the existing state-of-the-art NRKNet. Compared with methods designed with specific auxiliary input structures, such as KPAC, the present invention also achieves a significant performance improvement of 0.042 on FSIM. It is worth mentioning that the present invention remains competitive in indicators such as LPIPS, MUSIQ, FSIM and CKDN. These indicators better reflect human visual perception, showing that the deblurred images generated by the present invention are more visually attractive. PSNR and SSIM are two indicators specifically used to quantify pixel-level differences in the image domain, which further highlights the significant potential of the present invention in the single image defocus deblurring task.

[0087] In addition, in order to explore the impact of different parameter configurations, the core part of the model of the present invention integrates different types of ViT architectures; the main difference between ViT-B and ViT-T is the dimension of the hidden layer, which is crucial to the word segmentation process; ViT-B converts 16x16x3 image blocks into 768-dimensional vectors, so it can encapsulate information more effectively; in contrast, ViT-T converts the same image blocks into 192-dimensional vectors, which may cause information loss; according to the quantitative analysis of the present invention, since ViT-B has a larger dimension in the hidden vector and can retain more information, its performance is significantly better than ViT-T. However, even using ViT-T as the backbone network, the present invention has shown competitiveness comparable to other advanced methods in the defocus deblurring task.

[0088] exist Figure 2 In the paper, the present invention presents three cases, comparing the effects of five different methods with our method. In the first case, the present invention effectively restores the content structure of the image, while other methods usually leave significant background blur; in the deblurring effect, KPAC and DPDNet introduce some artifacts, especially in the edge areas of the image, which is particularly obvious in the second case; in contrast, the present invention successfully reconstructs the image structure, makes the edges sharper, and produces clear and visually pleasing images; these results highlight the advantages of the present invention in understanding the image context, and can more accurately restore details, reduce artifacts, and accurately handle defocused areas.

[0089] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A defocus deblurring method based on a hierarchical model, characterized in that: Specifically include: Step 1: Use the basic feature encoder to transform the blurred image x∈R C×H×W Split into h∈R N×D ; Step 2: Context Encoder Use Learn clearer and more abstract representations h s ; Step 3: The decoder will and h s As input, reconstruct the deblurred image 2. The defocus deblurring method based on a hierarchical model according to claim 1, characterized in that: In step 2, the context encoder uses Learn clearer and more abstract representations h s Specifically include: Using a linear layer from Prediction s : In formula (1), W represents the weight of the linear layer; Reference Features Obtained by applying the same encoder to the reference image y, the dimension of the reference image y is R C×H×W ; The introduction of the classification loss function based on the second-level cross entropy promotes and Learn features that distinguish blurry from sharp images: In formula (2), t represents the label, and t∈{0,1}, 0 represents blur, 1 represents clear, and N represents the tensor h * The total number of elements in , σ represents the sigmoid function, which is used to normalize the eigenvalue interval, and the eigenvalue interval is (0,1); The overall loss of the blur-to-sharp converter module is defined as: In formula (3), d(·) is used to calculate the distance between representations, and the hyperparameter λ is used to balance the importance of classification loss and other loss components; By using the loss function l BST The context encoder is trained to obtain the representation related to ambiguity and predict the s Clear representation of representation.

3. The defocus deblurring method based on a hierarchical model according to claim 2, characterized in that: Introducing a joint embedding architecture in the context encoder to enhance the representation h s The architecture includes an information content preservation module, which uses the regularization standard defined by VICReg to remove details irrelevant to the blur. The information content preservation module maps the representation to an embedding space through a multi-layer perceptron and derives a regularization loss from it. Regularization losses are used in the information content preservation module. These losses are based on three principles: variance, invariance, and covariance. When calculating the regularization loss, the information content preservation module receives two sets of inputs: a batch of reference representations h r1 and the representation h converted from the fuzzy-sharp converter s , through MLPf θ (·) Embed the above representation into a low-dimensional space to enhance the abstraction capability: In formula (4), e(·) represents the feature average operator, The dimension is R D , excluding the [CLS]| token used for classification in the transformer architecture when calculating the average; two sets of batch embeddings are derived from the mapping, denoted as and Where m is the batch size; The regularized loss function of the information content preserving module is defined as: In formula (5), z represents or γ and ε represent hyperparameters, represents the mean of the batch embedding, i.e. The total loss function of the information content preservation module contains three regularization terms, positioned as: In formula (6), μ1 and μ2 are hyperparameters used to balance the impact of various loss terms.

4. The defocus deblurring method based on a hierarchical model according to claim 1, characterized in that: The decoder uses the basic feature encoder and the context encoder to extract and h s Generate deblurred image The decoder is trained using the traditional l1 loss function: For the entire model, the final loss is defined as: In formula (8), α and β are parameters that balance different loss terms.

5. The defocus deblurring method based on a hierarchical model according to claim 4, characterized in that: The training strategy for training the decoder is as follows: the base feature encoder parameterized by the pre-trained MAE is frozen during the training of the context encoder, and then this model is trained on the large-scale ImageNet dataset; widely accepted data augmentation techniques are used, including random cropping to a size of 224×224, random scaling from 0.2 to 1.0, and random horizontal flipping; the loss of the entire model is minimized according to the loss function defined in formula (8), and the context encoder is initialized using the parameters of the pre-training stage.

Images