Neural network light field image deblurring method based on multi-head cross attention mechanism

Abstract

This invention belongs to the field of image deblurring technology and discloses a neural network light field image deblurring method based on a multi-head cross-attention mechanism. Specifically, the network is trained on a LFDOF dataset generated by a light field; the network is fine-tuned using feature loss on a DPDD dataset collected by a secondary method to mitigate the difference between the defocus blurs present in the two domains; the features of the two defocus blur datasets, LFDOF and DPDD, are analyzed to develop a new deblurring training strategy based on a multi-head cross-attention mechanism; and an end-to-end network architecture is proposed, equipped with a novel dynamic residual block, to reconstruct a clear image in a coarse-to-fine manner. The end-to-end neural network designed in this invention can effectively eliminate spatially varying defocus blurs, solving the problem that simple secondary shooting cannot achieve pixel-level correspondence between defocused and in-focus image pairs.

Description

Technical Field

[0001] The present invention belongs to the technical field of image deblurring, and in particular relates to a neural network light field image deblurring method based on a multi-head cross attention mechanism. Background Art

[0002] Using a wide aperture increases light throughput, enabling images to be captured with shorter exposure times. However, this also reduces the depth of field (DOF), meaning only points close to the focal plane are captured sharply, while points further away are projected onto the camera sensor as separate image points, a phenomenon known as defocus blur. This blur is referred to as the circle of confusion. Shallow depth of field is sometimes a desirable aesthetic effect for photographers, but it can also degrade important visual information. Therefore, recovering a fully focused image from a defocused version is desirable to reveal underlying information and facilitate artificial intelligence applications. While this approach holds great potential, defocus deblurring remains a challenging problem due to its spatially varying nature, where each point has its own circle of confusion diameter, depending on the depth of the corresponding scene point. Furthermore, the shape of the circle of confusion varies with its relative position to the optical axis. To address defocus blur, the most intuitive approach is a two-step approach: first estimating a blur kernel for each pixel and then applying non-blind deconvolution. However, both steps have limitations. First, the estimation of blur kernel is often inaccurate based on simple Gaussian or disk kernel assumptions; second, even if an accurate blur kernel is given, deconvolution will introduce ring artifacts at the edges due to the Gibbs phenomenon.

[0003] Recently, researchers have adopted end-to-end deep neural networks to directly restore sharp images from defocus blur, which greatly outperforms the traditional two-step method in terms of performance and efficiency; these networks are trained on a dataset called Dual Pixel Defocus Deblurring (DPDD), which is continuously captured using different aperture sizes to obtain defocused and fully focused image pairs; however, it is almost impossible to capture defocused and fully focused image pairs with accurate correspondence in two photos, especially for outdoor scenes due to moving objects (such as plants, cars) and illumination changes: taking the DPDD dataset as an example, since the dataset is captured using different aperture sizes, there is a misalignment problem between the defocused and fully focused image pairs due to the two shots, and there will be pixel-wise loss. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a neural network light field image deblurring method based on a multi-head cross-attention mechanism, constructs a defocus blur dataset LFDOF generated by light field and a DPDD dataset collected by a secondary method, generates a large number of defocus images with various degrees of freedom and focal lengths from a single light field sample to train the constructed neural light field network to obtain image pairs with corresponding relationships, and then uses feature loss to fine-tune the network to reduce the difference between the defocus blur in the two domains; an end-to-end deep neural network is used to directly restore a clear image from the defocus blur.

[0005] The neural network light field image deblurring method based on the multi-head cross attention mechanism described in the present invention comprises the following steps:

[0006] Step 1: Construct the LFDOF dataset, the DPDD dataset, and the test set; the LFDOF dataset is used as the training set, and the DPDD dataset is used as the auxiliary network fine-tuning set;

[0007] Step 2: Construct a neural light field network model based on a multi-head cross-attention mechanism and establish a deblurring training strategy for the neural light field based on the multi-head cross-attention mechanism;

[0008] Step 3: Use the LFDOF dataset to train the neural light field network model based on the multi-head cross-attention mechanism to obtain corresponding defocused and all-focused image pairs; use the DPDD dataset to fine-tune the network using feature loss to obtain the trained neural light field network model;

[0009] Step 4: Use the trained neural light field network model to process and test the test set containing blurred images to obtain clear test images after processing.

[0010] Furthermore, the neural light field network model includes an encoder, a decoder and several cascaded dynamic residual blocks; the input of the neural light field network model is the blurred image x in the dataset LFDOF and the dataset DPDD LFDOF and x DPDD , multi-scale pyramid features are extracted by the encoder ε, and then added to the corresponding scale of the decoder D through the skip connection; several dynamic residual blocks are added to each scale of the decoder D, and the input of several dynamic residual blocks is the blurred image x from the dataset LFDOF LFDOF And the decoder D outputs data of different scales, and the output is the deblurred image.

[0011] Furthermore, the dynamic residual block includes a multi-head cross attention mechanism module and a multi-layer perceptron;

[0012] The input features of each multi-head cross-attention mechanism module are tokenized and reshaped into a sequence of flattened two-dimensional patches of size P and P / 2 respectively; the patches are mapped to the same area of ​​the encoder features of their corresponding scale; through this process, the size of the original channel is maintained;

[0013] The labels of these two input features As the key, the value T ∑ =Concat(T1,T2); These tokens are fed into the multi-head cross attention mechanism module and the residual structure of the multi-layer perceptron to encode channels and dependencies and refine the features Q from each encoder i , each encoder uses multi-scale features;

[0014] The multi-head cross attention mechanism module contains three inputs (Q1, Q2, C ∑ ), including two markers T i as the query and a concatenated token T ∑ as keys and values;

[0015] Q i =T i W Q ,K=T Σ W K ,V=T Σ W V

[0016] in is the weight of different inputs, d is the sequence length, C i (i=1,2) are the sizes of the two input channels, C ∑ =Concat(C1,C2);Q i is the feature of each encoder, V and K are two weighted parameters involved in the calculation;

[0017] because In the above process, a similarity matrix is ​​generated And weight V through the cross-attention CA mechanism;

[0018]

[0019] where ψ(·) and σ(·) represent the instance normalization function and the softmax function, respectively; the superscript T is the transposition marker;

[0020] Attention is performed along the channel axis, and instance normalization is used. In the case of N-head attention, the output of the multi-head cross attention mechanism module is calculated as follows:

[0021]

[0022] Where N is the number of inputs; after that, an MLP and residual operator are applied, and the output obtained is as follows:

[0023] O i =MCA i +MLP(Q i +MCA i )

[0024] For simplicity, the layer normalization LN in the equation is omitted; the operation in the above formula is repeated L times to construct an L-layer transformer; the two outputs O1 and O2 of the Lth layer are reconstructed through an upsampling operation and then reconstructed through the convolution layer to obtain the output of the residual block.

[0025] Furthermore, the deblurring training strategy of the neural light field based on the multi-head cross attention mechanism is as follows:

[0026] First, a neural light field network model is trained on the light field generation dataset LFDOF to obtain corresponding defocus and all-focus image pairs; then, the neural light field network model is fine-tuned using the feature loss on the dual-lens method collection dataset DPDD to evaluate the difference between defocus blur in the two domains.

[0027] Furthermore, a neural light field network model is trained on the LFDOF dataset to obtain corresponding defocused and in-focus image pairs, specifically:

[0028] The constructed neural light field network model is trained as a mapping function F parameterized by θ, thereby transforming the input LFDOF image x LFDOF Restore to clear image The mapping function is expressed as:

[0029]

[0030] The image restored by the neural light field network model is obtained by adjusting the loss function to optimize θ, so that the image restored by the neural light field network model Minimize the distance between the value y and the ideal clear image;

[0031]

[0032] Among them, (x i ,y i ) is a pair of defocused and in-focus images; θ is the optimized value of the adjusted loss function.

[0033] Furthermore, the network is fine-tuned using feature loss using the dataset DPDD to obtain a trained neural network light field model, specifically:

[0034] During the training phase, use Norm:

[0035]

[0036] in is the sparse regular operator of the dataset LFDOF, and y LF They are the clear images output under LFDOF of the dataset and the clear images under theoretical ideal conditions;

[0037] In the fine-tuning stage, a VGG-based feature loss is applied to transfer the learned knowledge to the target domain in the feature space. This loss is recorded as

[0038]

[0039] in It is the feature loss operator of the dataset DPDD based on VGG, is the instance normalization function, and y DPDD They are the clear images output under the dataset DPDD and the clear images under theoretical ideal conditions.

[0040] The beneficial effects described in the present invention are as follows: the present invention analyzes the features of two defocus blur datasets LFDOF and DPDD through training, and develops a new single image defocus deblurring training strategy; proposes an end-to-end network architecture, which is equipped with a novel dynamic residual block, and connects several dynamic residual blocks to each decoder in a cascade manner, and reconstructs a clear image in a coarse-to-fine manner, which can effectively eliminate spatially varying defocus blur, solves the problem that simple secondary shooting cannot achieve pixel-level correspondence between defocused and full-focus image pairs, and improves performance and efficiency; the present invention introduces a module based on a multi-head cross-attention mechanism, which uses shared convolution kernels to extract common information and uses private kernels to extract modality-specific information. Through the multi-head cross-attention mechanism and multi-scale channels, the semantic and resolution gap between low-level and high-level features is bridged to capture more complex channel correlations. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flow chart of the method of the present invention;

[0042] Figure 2 It is the overall framework of the algorithm model in the embodiment of the present invention;

[0043] Figure 3 This is a framework for a feature extraction module based on a multi-head cross-attention mechanism proposed in this paper;

[0044] Figure 4 The specific internal structure of the feature extraction module of the multi-head cross attention mechanism;

[0045] Figure 5 A visual comparison diagram of the network trained on LFDOF only, DPDD only, and both datasets simultaneously; DETAILED DESCRIPTION

[0046] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0047] like Figure 1 As shown, the neural network light field image deblurring method based on the multi-head cross attention mechanism of the present invention comprises the following steps:

[0048] Step 1: Construct the dataset LFDOF, dataset DPDD, and test sets CUHK and PixelDP;

[0049] Step 2: Construct a neural light field network model based on a multi-head cross-attention mechanism and establish a deblurring training strategy for the neural light field based on the multi-head cross-attention mechanism;

[0050] Step 3: Use the LFDOF dataset to train the neural light field network model based on the multi-head cross-attention mechanism to obtain corresponding defocused and all-focused image pairs; use the DPDD dataset to fine-tune the network using feature loss to obtain the trained neural light field network model;

[0051] Step 4: Use the trained neural light field network model to process and test the test set containing blurred images to obtain clear test images after processing.

[0052] In step 1, a LFDOF dataset is generated as the main training set: this dataset uses light field synthetic aperture and refocusing technology to generate a set of real defocused and fully focused image pairs depicting various natural scenes for network training;

[0053] The DPDD dataset is generated as an auxiliary network fine-tuning set: This dataset leverages the available data of dual-pixel (DP) sensors on most modern cameras; DP sensors are used to assist the camera's autofocus by capturing two sub-aperture views of a scene in a single image shot; these two sub-aperture images are used to compute the appropriate lens position to focus on a specific scene region. Each scene has: (i) a defocused image captured at a large aperture; (ii) two associated DP sub-aperture views; and (iii) a corresponding fully focused image captured with a small aperture.

[0054] like Figure 2 As shown, the neural light field network model includes an encoder, a decoder and several cascaded dynamic residual blocks; the input of the neural light field network model is the blurred image x in the dataset LFDOF and the dataset DPDD LFDOF and x DPDD , multi-scale pyramid features are extracted by the encoder ε, and then added to the corresponding scale of the decoder D through the skip connection; several dynamic residual blocks are added to each scale of the decoder D, and the input of the dynamic residual block is the blurred image x from the dataset LFDOF LFDOF And the data of different scales output by the decoder D, the output of the previous dynamic residual block is used as one of the two inputs of the next residual block, until the output of the last residual block is used as the final output clear image.

[0055] like Figure 3 As shown, the input of the dynamic residual block is normalized, and a multi-head cross attention mechanism module and a multi-layer perceptron are introduced. The normalization is performed before adding the multi-layer perceptron, as follows:

[0056] The input features of each module are tokenized and reshaped into a sequence of flattened two-dimensional patches of size P, P / 2 respectively; the patches are mapped to the same area of ​​the encoder features of their corresponding scale; through this process, we maintain the size of the original channel;

[0057] The labels of these two input features As the key, the value T ∑ =Concat(T1,T2); These tokens are fed into the multi-head cross attention mechanism module and the residual structure of the multilayer perceptron to encode channels and dependencies, thus refining the features Q from each encoder i , each encoder uses multi-scale features;

[0058] like Figure 4 As shown, the multi-head cross attention mechanism module contains 3 inputs (Q1, Q2, C ∑ ), including two markers T i as the query and a concatenated token T∑ as keys and values;

[0059] Q i =T i W Q ,K=T ∑ W K ,V=T ∑ W V

[0060] in is the weight of different inputs, d is the sequence length, C i (i=1,2) are the sizes of the two input channels, C ∑ =Concat(C1,C2), concat is a function used to concatenate two or more feature maps in the channel or num dimension; Q i is the feature of each encoder, V and K are two weighted parameters involved in the calculation;

[0061] because In the above process, a similarity matrix is ​​generated And weight V through the cross attention (CA) mechanism;

[0062]

[0063] where ψ(·) and σ(·) represent instance normalization function and softmax function, respectively. The former is a private convolution kernel used to extract modality-specific information, while the latter is a shared convolution kernel used to extract common information. The superscript T is the transposition marker.

[0064] Attention is performed along the channel axis, and instance normalization is used. In the case of N-head attention, the output of the multi-head cross attention mechanism module is calculated as follows:

[0065]

[0066] Where N is the number of heads (inputs); after that, an MLP and residual operator are applied, and the output obtained is as follows:

[0067] O i =MCA i +MLP(Q i +MCA i )

[0068] For simplicity, we omit the layer normalization LN in the equation; repeat the operation in the above formula L times to build an L-layer transformer; construct an L-layer transformer; the two outputs O1 and O2 of the Lth layer are reconstructed by an upsampling operation and then reconstructed by the convolution layer to obtain the output of the residual block.

[0069] The specific implementation of step 3 is as follows: the neural light field network model based on the multi-head cross attention mechanism is trained using the dataset LFDOF to obtain defocused and all-focused image pairs with corresponding relationships:

[0070] The constructed deep learning network model is trained as a mapping function F parameterized by θ, thereby transforming the input LFDOF image x LFDOF Restore to clear image The mapping function is expressed as:

[0071]

[0072] The image is restored by the network model. By adjusting the loss function to optimize θ, the image restored by the neural light field network model is Minimize the distance to the ideal clear image;

[0073]

[0074] Where (x i ,y i ) is a pair of defocused and in-focus images; θ is the optimized value of the adjusted loss function.

[0075] Using the dataset DPDD, we fine-tune the network using feature loss to obtain a trained neural network light field model:

[0076] The defocus blur produced by traditional digital cameras and light field cameras is different. To bridge this gap, we propose a training strategy that exploits the intensity and overcomes the shortcomings of light field data. Specifically, we apply the light field generation dataset LFDOF in the main training round to obtain highly accurate image correspondences; then, we fine-tune the network using the DPDD dataset to alleviate the difference between the two domains. Different losses are used in each stage. In the main training stage, Norm:

[0077]

[0078] In the fine-tuning stage, pixel loss should be avoided due to the misalignment between the defocused and in-focus pairs in the DPDD dataset (caused by the two shots). We apply a VGG-based feature loss in this step to transfer the learned knowledge to the target domain in the feature space, thus avoiding the exact matching in the image space. This loss is denoted as

[0079]

[0080] in It is the feature loss operator of the dataset DPDD based on VGG, is the instance normalization function, and y DPDD They are the clear images output under the DPDD dataset and the clear images under the theoretical ideal case. Note that this loss is applicable to all scales.

[0081] We compared the results of training the network only on LFDOF or DPDD datasets with those of training the network on LFDOF and then fine-tuning on DPDD. The latter greatly outperformed the former (as shown in Table 1) and produced the best quantitative (PSNR increased by 11.57% and 4.23%) and qualitative results (as shown in Table 2). Figure 5 shown).

[0082] Table 1

[0083] Training dataset PSNR↑ SSIM↑ LPIPS↓ LFDOF 23.076 0.698 0.378 DPDD 24.700 0.744 0.337 LFDOF&DPDD 25.745 0.711 0.257

[0084] The above description is only a preferred embodiment of the present invention and is not intended to further limit the present invention. All equivalent changes made using the contents of the present invention description and drawings are within the scope of protection of the present invention.

Claims

1. A neural network light field image deblurring method based on a multi-head cross-attention mechanism, characterized by: The following steps are involved: Step 1: Construct the LFDOF dataset, DPDD dataset, and test set; The dataset LFDOF is used as the training set, and the dataset DPDD is used as the auxiliary network fine-tuning set; Step 2: Construct a neural light field network model based on a multi-head cross-attention mechanism and establish a deblurring training strategy for the neural light field based on the multi-head cross-attention mechanism; Step 3: Using the LFDOF dataset, the neural light field network model based on the multi-head cross-attention mechanism is trained to obtain defocused and all-focused image pairs with corresponding relationships; Using the dataset DPDD, we fine-tune the network using feature loss to obtain a trained neural light field network model. Step 4: Use the trained neural light field network model to process the test set containing blurred images to obtain clear test images; The neural light field network model includes an encoder, a decoder and several cascaded dynamic residual blocks; the input of the neural light field network model is the blurred image x in the dataset LFDOF and the dataset DPDD. LFDOF and x DPDD , multi-scale pyramid features are extracted by the encoder ε, and then added to the corresponding scale of the decoder D through the skip connection; several dynamic residual blocks are added to each scale of the decoder D, and the input of several dynamic residual blocks is the blurred image x from the dataset LFDOF LFDOF And the data of different scales output by the decoder D is the deblurred image; The dynamic residual block includes a multi-head cross attention mechanism module and a multi-layer perceptron; The input features of each multi-head cross-attention mechanism module are tokenized and reshaped into a sequence of flattened two-dimensional patches with sizes P and P / 2 respectively; the patches are mapped to the same area of ​​the encoder features at their corresponding scales; Through this process, the dimensions of the original channel are maintained; The input feature label T i, As the key, i = 1, 2; value T ∑ =Concat(T1,T2); These tags are fed into the multi-head cross attention mechanism module and the residual structure of the multi-layer perceptron to refine the features Q from each encoder i , each encoder uses multi-scale features; The multi-head cross attention mechanism module contains three inputs (Q1, Q2, C ∑ ), also contains the input feature marker T i as the query, and the value T ∑ as keys and values; Q i =T i W Q ,K=T ∑ W K ,V=T ∑ W V in is the weight of different inputs, d is the sequence length, C i is the size of the two input channels, i=1,2, C ∑ =Concat(C1,C2);Q i is the feature of each encoder, V and K are two weighted parameters involved in the calculation; because Generates a similarity matrix And weight V through the cross-attention CA mechanism; where ψ(·) and σ(·) represent the instance normalization function and the softmax function, respectively; the superscript T is the transposition marker; Attention is performed along the channel axis, and instance normalization is used. In the case of N-head attention, the output of the multi-head cross attention mechanism module is calculated as follows: Where N is the number of inputs; after that, an MLP and residual operator are applied, and the output obtained is as follows: O i =MCA i +MLP(Q i +MCA i ) The layer normalization LN in the equation is omitted; the operation in the above formula is repeated L times to construct an L-layer transformer; the two outputs O1 and O2 of the Lth layer are reconstructed through an upsampling operation and then reconstructed through the convolution layer to obtain the output of the residual block.

2. The neural network light field image deblurring method based on multi-head cross attention mechanism according to claim 1 is characterized in that: The deblurring training strategy of the neural light field based on the multi-head cross attention mechanism is as follows: First, a neural light field network model is trained on the light field generation dataset LFDOF to obtain corresponding defocus and all-focus image pairs; then, the neural light field network model is fine-tuned using the feature loss on the dual-lens method collection dataset DPDD to evaluate the difference between defocus blur in the two domains.

3. The neural network light field image deblurring method based on multi-head cross attention mechanism according to claim 2 is characterized in that: The neural light field network model is trained on the LFDOF dataset to obtain defocused and all-focused image pairs with corresponding relationships, specifically: The constructed neural light field network model is trained as a mapping function F parameterized by θ, thereby transforming the input LFDOF image x LFDOF Restore to clear image The mapping function is expressed as: The image is restored by the neural light field network model, and the loss function is optimized by adjusting θ to make the restored clear image Minimize the distance to the ideal clear image; Among them, (x i ,y i ) is a pair of defocused and in-focus images; θ after adjusting the loss function optimization.

4. The neural network light field image deblurring method based on multi-head cross attention mechanism according to claim 3 is characterized in that: Using the dataset DPDD, we fine-tune the network using feature loss to obtain a trained neural network light field model, specifically: During the training phase, the l1 norm is used: in is the sparse regular operator of the dataset LFDOF, and y LF They are the clear images output under LFDOF of the dataset and the clear images under theoretical ideal conditions; In the fine-tuning stage, a VGG-based feature loss is applied to transfer the learned knowledge to the target domain in the feature space. This loss is recorded as in It is the feature loss operator of the dataset DPDD based on VGG, is the instance normalization function, and y DPDD They are the clear images output under the dataset DPDD and the clear images under theoretical ideal conditions.

Images