A defocus map estimation method based on adaptive region feature and attention fusion

Abstract

The application provides an out-of-focus image estimation method based on region feature adaptation and attention fusion, which uses a common Encoder-Decoder structure as a basic network for an image restoration task. Secondly, a skip connection is added in the basic network, and a channel attention mechanism is introduced in the decoder, so that the decoder can better retain and utilize the feature information extracted in the encoder. At the same time, a region feature adaptation network is used for image-to-image regression, which is attached to the basic network to minimize the domain difference between the synthesized features and the real features, and a channel attention mechanism is also introduced in the region feature adaptation network to further enhance the representation ability of the network. Finally, during training, the region feature adaptation network is first trained as a discriminator, and the discriminator loss is used to classify the features of the synthesized domain and the real domain, and then the basic network is trained to minimize the domain difference between the synthesized and real out-of-focus image features.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to defocus image estimation technology based on deep learning. Background Technology

[0002] In photography, the reflection and refraction of light cause a certain degree of blurring when the image is formed; this blurred image is called a defocus image. Estimate the amount of blur in the defocus image, and with this information, you can adjust the focus or angle in post-processing to make the image sharper.

[0003] Typical defocus map estimation methods first detect edges from a blurred image, then measure the degree of blur around the edges, and finally determine the amount of blur in a uniform region through interpolation. However, previous edge-driven methods have several limitations. First, since edges in a blurred image are often also blurred, their detection results are inaccurate. Second, blur estimation at edges is inherently prone to error because pixels on the boundaries of objects with depth discontinuities often contain varying degrees of blur mixing. Third, the instability of edge blur estimation leads to unreliable predictions in uniform regions; the amount of blur obtained solely by relying on features defined around the image edges is often inaccurate. In other words, the estimated degree of blur for different parts of an object boundary may be irrelevant, and interpolating them into the interior of a uniform object only produces smooth but less accurate blur estimates. For example, the blur estimate for an object with a single depth may not be constant because the degree of blur measured relative to the edge individually may differ when the edge has different depth discontinuities with nearby objects.

[0004] With the significant achievements of deep learning in various computer vision tasks, deep learning-based methods have also been widely applied to defocus image estimation. While deep learning methods have shown great promise in addressing image degradation, defocus image estimation requires a large amount of accurate training data, consisting of pairs of full-focus and defocused images, which is difficult to collect. This is because the two images should be captured under identical exposure and lighting conditions to ensure pixel-level correspondence between the image pairs. However, limitations such as varying lighting and camera hardware often lead to inconsistent exposure and lighting conditions with ordinary cameras under different focus settings, thus affecting the quality of the image pairs. This means it is difficult to obtain a large number of real-world paired samples to train a defocus image estimation network with good generalization ability. As research has progressed, many researchers have used synthetic blur datasets as a compromise.

[0005] As a compromise, although the image quality of synthetic blur datasets has improved significantly thanks to the joint efforts of many scholars, there are still significant differences between them and the features of real blurry scenes. This leads to a serious impact on the performance of defocus map estimation networks trained on synthetic blur datasets in real defocus blurry scenes, with performance indicators showing a clear downward trend. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a method that helps to output realistic defocus estimation maps, addressing the issue that the number of real blurry dataset images is insufficient in existing defocus map estimation methods, and that the pixel and structural information contained in low-resolution images is not fully utilized.

[0007] The technical solution adopted by this invention to solve the above-mentioned technical problems is a defocus map estimation method based on region feature adaptation and attention fusion, comprising:

[0008] Training steps:

[0009] The training image set is input into the defocus estimation network, and the defocus estimation network outputs the blur amount of the image to the region feature adaptive network and cascaded residual blocks;

[0010] The training image set includes a training set of synthetic defocused images with labels, a training set of real defocused images without labels, and a training set of real defocused images with labels and binary blur. The training set of synthetic defocused images with labels belongs to the synthetic domain, while the training sets of real defocused images without labels and real defocused images with labels and binary blur belong to the real domain.

[0011] The region feature adaptive network classifies whether the blur amount of the input image comes from the real domain or the synthetic domain, and outputs the blur amount feature value of the real image or the blur amount of the synthetic image to the residual block of the cascaded convolutional layer.

[0012] The cascaded residual blocks output either a real defocus estimation image or a synthetic defocus estimation image, and the region feature adaptive network classifies the synthetic defocus estimation image into the synthetic domain before inputting it into the convolutional layer network.

[0013] The accuracy of the defocused image estimation is determined by the output of the convolutional layer network.

[0014] The defocus estimation network predicts the amount of blur in the input image and uses the amount of blur in the output image that confuses the synthetic and real domains as the training objective.

[0015] The region feature adaptive network is trained to correctly classify the amount of blur in images from the synthetic and real domains, thereby training the defocus estimation network to minimize the domain difference between features from the synthetic and real domains.

[0016] The convolutional network is trained to correctly classify whether each pixel of the synthetic defocused image has the correct amount of blur, thereby training the defocused estimation network to estimate the correct sharpness from the domain-transferred features.

[0017] Test steps:

[0018] The image to be processed is input into the defocus estimation network, and the defocus estimation network outputs the amount of blur in the image to complete the defocus map estimation.

[0019] Specifically, this invention designs an end-to-end trained defocus image estimation network, RFAAF-DME: First, this invention adopts the common Encoder-Decoder structure as the base network for image restoration tasks. The encoder part uses a pre-trained VGG19 network to progressively extract features from the input defocused image. The decoder part adopts a U-Net structure, fusing encoder features with low-level features and upsampling the encoder output feature map to the size of the original input image. Second, skip connections are added to the base network, and a channel attention mechanism is introduced in the decoder, enabling the decoder to better retain and utilize the feature information extracted by the encoder, improving the detail fidelity of image restoration. Simultaneously, a region feature adaptation network is used for image-to-image regression, appended to the base network to minimize the domain difference between synthetic and real features. A channel attention mechanism is also introduced in the region feature adaptation network to further enhance the network's representational ability, reduce computation on unimportant regions, save computational resources, and accelerate the network. Finally, during training, this invention first trains a region feature adaptive network as a discriminator, uses the discriminator loss to classify the features of the synthetic and real domains, then trains the base network to minimize the domain difference between synthetic and real defocused image features, and uses the mean squared error (MSE) and mean absolute error (MAE) of the defocus estimation map loss function to estimate the overall structure of the defocus map and densely predict the amount of blur in the region.

[0020] Based on the above analysis of the shortcomings of existing defocus image estimation algorithms, this invention focuses on the following points when designing the network: (1) In order to solve the problem of insufficient number of real blurry dataset images, an improved approach of adding synthetic blurry datasets for training is adopted. At the same time, the regional feature adaptive method is considered to reduce the gap between the synthetic training images and real images and improve the generalization ability of the model. (2) Furthermore, attention is paid to effectively utilizing the pixel, structure and other information contained in the input low-resolution image. The feature representation in the defocused blurry image is fully extracted through the structure of hierarchical feature extraction, and the positional information of the defocused area is reasonably utilized through the attention mechanism. (3) Further, a loss function is designed to help generate realistic output images.

[0021] The beneficial effects of this invention are as follows:

[0022] (1) A defocus image estimation network based on encoder-decoder structure was built, and skip connections and channel attention mechanisms were added to enable the network to fully extract and utilize the effective information in the input defocus image.

[0023] (2) In the basic network for defocus map estimation, a region feature adaptive network that can minimize the features between the real defocused blurred image and the synthetic defocused blurred image is added to construct a defocus map estimation network with good performance.

[0024] (3) A loss function was designed that is conducive to preserving texture and details. Attached Figure Description

[0025] Figure 1 A schematic diagram of the backbone network and hop connection network structure;

[0026] Figure 2 The overall network structure of RFAAF-DME;

[0027] Figure 3 It is composed of regional feature adaptive networks;

[0028] Figure 4 This is a schematic diagram of the channel attention structure;

[0029] Figure 5 Quantitative comparison with classic defocus map estimation networks on the CUHK and DED datasets;

[0030] Figure 6 To provide a qualitative comparison with classic defocus image estimation networks on the CUHK dataset;

[0031] Figure 7 To provide a qualitative comparison with classic defocusing image estimation networks on the DED dataset. Detailed Implementation

[0032] In this example, the input defocused image is resized to 224×224 and then input into the defocus image estimation network, ultimately generating a defocused estimated image of the same size as the input image.

[0033] The defocusing map estimation network consists of an initialization layer and a backbone network connected in sequence.

[0034] The initialization layer is used to adjust the input defocused and blurred image to obtain a 224×224 feature map.

[0035] The backbone network is an encoder-decoder structure with added skip connections, consisting of N+1 encoder layers and N decoder layers. The N+1 encoder layers are connected from high to low resolution to form a downsampling structure, with each encoder layer being a downsampling block. The N decoder layers are connected from low to high resolution to form an upsampling structure, with each decoder layer being an upsampling block. The last decoder layer restores the output feature map to a defocused map to estimate the input image size of the network.

[0036] On one hand, the feature map output by the downsampling block and the feature map of the same resolution output by the upsampling block are concatenated for the first feature fusion. On the other hand, each upsampling module inputs its output feature map into the attention mechanism for processing. The feature map output by the attention mechanism is then concatenated with the skip connection features for the second feature fusion. The feature map after the second feature fusion is further reduced in dimensionality and used as the input to the next upsampling block or the output of the final defocus map estimation network. During the skip connection process, a channel attention module is inserted to select features that are helpful in estimating the defocus map.

[0037] like Figure 1 The encoder-decoder structure shown includes a fifth downsampling block, a fourth downsampling block, a third downsampling block, a second downsampling block, a first downsampling block, a first upsampling block, a second upsampling block, a third upsampling block, and a fourth upsampling block connected in sequence. The feature map gradually reduces its resolution to 14×14 through 5 downsampling blocks, and then gradually increases its resolution through 4 upsampling blocks, generating a feature map of the same size as the initial feature map.

[0038] Building upon the backbone network, a region feature adaptive network is added. This network utilizes a generative adversarial loss function to minimize the features between the real and synthetic defocused blurred images, ensuring that both regions have the same distribution of extracted features. Finally, the results are processed through cascaded residual blocks to generate the final output.

[0039] The Region Feature Adaptive Network (RIN) is a cascaded three-layer network consisting of convolutional layers, batch normalization layers, and a channel attention module. Essentially, it's a CNN with three convolutional layers, each followed by a batch normalization layer and a Leaky ReLU activation unit. Inserting the channel attention module further enhances the network's representation capabilities, reduces computation on unimportant regions, saves computational resources, and accelerates the network.

[0040] During the training of the defocus map estimation network, adversarial training is used to stabilize the training.

[0041] The following embodiments provide a detailed description of the various details involved in implementing the present invention.

[0042] During the training of the defocus map estimation network B, a region feature adaptive network D, a cascaded structure of multiple residual blocks, and a 1×1 CNN structure are introduced to form a convolutional network S.

[0043] like Figure 2 As shown, during training, the real defocused blurred image and the synthesized defocused blurred image are input into the defocus estimation network B, respectively. The defocus estimation network outputs the blur amount of the real image and the blur amount of the synthesized image to the region feature adaptive network D. After minimizing the domain difference between the synthesized features and the real features, the region feature adaptive network D outputs the blur amount feature value of the real image and the blur amount of the synthesized image to the cascaded residual block. The cascaded residual block outputs the real defocus estimated image and the synthesized defocus estimated image. The synthesized defocus estimated image is then input into the convolutional layer network S, and the convolutional layer network S outputs the accuracy of the synthesized defocus estimated image.

[0044] (1) Construct an encoder-decoder defocus map estimation network B with added skip connections, such as Figure 1 As shown:

[0045] The encoder-decoder architecture consists of 5 downsampling blocks and 4 upsampling blocks. First, the defocused image is resized to 224×224 pixels and input into the backbone network. It then undergoes five downsampling blocks to progressively reduce the resolution to 14×14 pixels, followed by four upsampling blocks to progressively increase the resolution, ultimately generating a defocused estimated image of the same size as the input image. The image sizes processed by the five downsampling blocks are: 224×224×64, 112×112×128, 56×56×256, 28×28×512, and 14×14×512. The image sizes processed by the four upsampling blocks are: 28×28×512, 56×56×256, 112×112×128, and 224×224×64. The first two data points of the image size represent the image resolution, and the last data point represents the number of channels.

[0046] Considering that different network layers focus on different features, such as shallow networks focusing on texture features and deep networks focusing on global features, and that both global and texture features are very important for defocused image estimation tasks, and that the loss of some edge features is unavoidable due to downsampling operations, this invention adds skip connections to the backbone network in order to utilize more comprehensive information from the defocused image. This involves concatenating feature maps of the same resolution in the encoder and decoder to achieve feature fusion.

[0047] (2) Add a channel attention module SE to the above defocusing image estimation network B.

[0048] In deep learning, encoders typically extract a large amount of feature information, but not all extracted features provide effective information for defocus image estimation. For example, the input defocused image may contain noise, artifacts, and other invalid information. Introducing all this information into the defocus image estimation network may lead to edge loss and other issues in the output image, resulting in poor image quality. Therefore, an attention mechanism is introduced to optimize this problem. The concept of attention mechanism in computer vision originates from the human brain's perception of the external environment; it helps neural networks focus on key information and ignore unimportant information. This characteristic allows the attention mechanism to reduce the computational load of the network and improve its efficiency. This invention adds a channel attention module (SE) to the feature map extracted from each upsampling block in the decoder to learn which features contribute to generating the defocus estimation map. Simultaneously, a channel attention module is also introduced into the region feature adaptation module to further optimize the model's performance, such as... Figure 2 As shown, the Channel Attention Module (SE) works by modeling the importance of each feature channel and enhancing or suppressing different channels based on the learning results, thereby improving the model's ability to extract key information.

[0049] like Figure 4 As shown, the input feature map is processed through a two-branch structure. The weight branch contains two key operations: Squeeze and Excitation. In the Squeeze operation, the input feature map (of size h×w×c) is globally averaged and pooled, compressing it into a 1×1×c feature map with the number of feature channels remaining unchanged, thus obtaining global information about the feature map. Next, the Excitation operation uses two fully connected layers. The first fully connected layer has relatively few neurons, only c / 16, while the second fully connected layer has the same number of neurons as the input channels. Through these two fully connected layer processes, the correlation between channels can be learned, generating weights for each feature channel. Then, a sigmoid activation function is used to fix the learned weight values ​​between 0 and 1, obtaining the channel weights. Finally, the channel weights are multiplied by the original feature map, thereby changing the importance of different channels and adaptively focusing on the information between the feature map channels.

[0050] (3) Add a region feature adaptive network D

[0051] Domain Feature Adaptation (DFA) is a machine learning method designed to address the generalization problem caused by discrepancies between training data and the target region. In many practical applications, datasets may only contain labeled data from the source domain, but the goal is to apply the model to the target domain, where there is a distributional difference between the source and target domains. DFA aims to improve model performance on the target region by utilizing source domain data, without requiring labeled data from the target region. This invention uses a region generation method for image-to-image regression to address the problem of limited datasets of real defocused image pairs. Leveraging the advantages of DFA, parameters are flexibly adjusted based on the characteristics and attributes of different regions in real and synthetic defocused image pairs, improving the performance and effectiveness of the defocus map estimation algorithm on images with varying degrees of defocus and complexity.

[0052] The region-adaptive network of this invention compares features of real and synthetic defocused images captured by a blur estimation network. An adversarial training strategy is used during training to ensure that the two regions have the same distribution in terms of extracted features. The region-adaptive network uses a discriminator from a GAN framework, with the ultimate goal of making the captured features of real and synthetic defocused images indistinguishable. Figure 3 As shown, the embodiment designs the region feature adaptive network as a CNN with four convolutional layers with 4×4 kernels and a stride of (2,2). The number of output channels of the four convolutional layers are 64, 128, 256, and 512 respectively. Each convolutional layer is followed by a batch normalization layer (BatchNorm) and a leaky rectified linear unit (ReLU) activation.

[0053] (4) Train the model and verify its effectiveness through experiments.

[0054] The constraints for the defocus map estimation network can be mainly divided into the following five points:

[0055] (a) Training objectives

[0056] The ultimate goal of this invention is to train a blur estimation network B to estimate the amount of blur in a real image and obtain a defocused image. To achieve this, three different training sets D are used during training. S D R and D B Joint training by θ B θ D and θ S The network consists of a parameterized fuzzy estimation network B, a region feature adaptive network D, and a convolutional layer network S.

[0057] It is a training set of synthetic defocused images with labels, where, It is the nth synthesized defocused image I S y n for The corresponding defocus image.

[0058] It is a training set of unlabeled real defocused images. I represents the nth unlabeled true defocused image. R .

[0059] It is a labeled binary fuzzy image b n True defocus image The training set, where b n Each pixel is marked as either sharp or blurry. For the nth labeled true defocused image I B b n For the nth true defocused image A binary fuzzy image.

[0060] The real defocused and blurred images during training are from D R and D B The synthesized defocused and blurred image comes from D. S .

[0061] Given a training dataset, following common practices in adversarial training, a loss function L is used. g Training θ B and θ S Using the loss function L d Training θ D .

[0062] Loss function L g :

[0063]

[0064] Where |D| represents the number of elements to be taken from set D. B L S and L adv These are the mean squared error loss, the sigmoid cross-entropy loss, and the adversarial loss, respectively. λ S and λ adv It is a hyperparameter for balancing the loss term.

[0065] Loss function L d :

[0066]

[0067] Where L D It is the discriminator loss, λ D It is Lg and L d The hyperparameters that balance the two domains are 1, which indicates that the domain is classified as the synthetic domain and 0 indicates that the domain is classified as the real domain.

[0068] During training, networks D and S influence network B differently depending on the input domain. In the case of synthesized blurred images from labeled defocus maps, S ,y>∈D S The fuzzy graph loss L is calculated using the mean square error (MSE) of the measurement. B Directly minimize the defocus map y and the defocus estimation image B(I) output by network B S The difference between ) and the defocused estimated image output by network B is the predicted defocus map.

[0069] A true defocused image with a binary blur map. B ,b>∈D B This is used to calibrate sharpness measurements from domain-transferred features. Under the supervision of b, the sigmoid cross-entropy loss function L used for sharpness calibration is... S The defocused image B(I) output by network S is guided by network S to network B. B Whether the classification has the correct amount of fuzziness is determined, and finally, the network B is calibrated to estimate the correct sharpness from the domain-transferred features.

[0070] Finally, I S ∈D S I B ∈D B and I R ∈D R Together, they are used to minimize the domain difference between features extracted from the synthetic image and features extracted from the real defocused image. Image I S From the synthesis domain, and I B and I R Derived from the real domain. This invention minimizes the discriminator loss L in an adversarial manner. D and adversarial loss L adv The network D is trained to correctly classify feature domains from different inputs, while the network B is trained to confuse D.

[0071] The following section details the mean squared error loss function, adversarial loss function, cross-entropy loss function, and discriminator loss function designed into the overall training loss function.

[0072] (b) Mean Squared Error Loss Function

[0073] The defocusing image estimation network B uses the mean squared error (MSE) loss function L during training. B This is used to estimate the overall structure of the defocused image and densely predict the amount of blur in the regions. Given a synthetic defocused image I of size W×H. S L​​B The definition is shown in the following formula:

[0074]

[0075] Where B(I) S ;θ B ) i,j It is a defocus estimation network with learning parameters θ B The predicted I at pixel (i,j) S fuzzy quantity, y i,j It is the value at pixel (i,j) in the defocused image.

[0076] L aux The auxiliary loss is proportional, defined as follows:

[0077]

[0078] Among them B l (I S ;θ B ,θ aux ) = A l (B l (I S ;θ B );θ aux ) is the output of level l sampled from network B, parameterized as θ aux Small auxiliary network A l Convert to a defocused image. Each auxiliary network A... l It consists of two convolutional layers, where the kernel number of the first layer varies with the layer number l. λ aux It is a balance parameter. W l ×H l y represents the size of the level l defocus image. l,i,j It is adjusted to W l ×H l Defocused image of size. L B The number of upsampling layers in B.

[0079] (c) Adversarial loss function

[0080] Next, this invention trains a defocus image estimation network to minimize the domain difference between synthetic and real defocus image features. Given a real defocus image I... R Domain-adaptive adversarial loss L adv As shown in the formula below:

[0081] L adv =-log(D(B) last (I R ;θ B ),θ D ))

[0082] Fixed parameter θ D Only train θ B The primary goal here is to train the defocus map estimation network so that it treats both real and synthetic defocused images as originating from the same domain. As region feature adaptation networks become increasingly capable of functioning as region classifiers, the defocus map estimation network needs to generate more features that cannot distinguish between real and synthetic regions in order to minimize inter-regional discrepancies in feature extraction.

[0083] (d) Cross-entropy loss function

[0084] The region feature adaptive network D focuses on modulating the overall distribution of features extracted from both real and synthetic defocused images, rather than specifically addressing the amount of blur in aligning corresponding features between two regions. In other words, the blur estimation network B cannot be directly applied to real defocused images because the amount of blur it acquires when learning from synthetic defocused images is not easily adaptable to real images. Therefore, the blur estimations for both domains need to be calibrated during training. To address this issue, additional information is provided to the features extracted from these images by associating the real defocused images output by network B with blur information from an available blur detection dataset. In this process, each pixel is labeled as sharp or blurred.

[0085] For a given synthetic defocused image in the dataset, this invention trains network S to classify the output of network B to check if the estimated blur is correct. A prediction is considered correct only if the estimated sharp pixels are located in sharp regions of the input image. A network S with 1×1 convolutional layers is constructed, and batch normalization and leaky ReLU layers are used after each convolutional layer to force network B to estimate blur densely.

[0086] Optimization is performed using sigmoid cross-entropy loss:

[0087]

[0088] Where b i,j Let be the value of the binary blurred image b at pixel (i,j), which is a true binary blurred map. During training, a 1×1 kernel is used to keep the receptive field sizes of networks B and S the same. Otherwise, as the receptive field of S increases, the gradients propagated from S to B will spread to regions with receptive fields larger than those of B.

[0089] (e) Discriminator Loss Function

[0090] This invention first trains a region feature adaptive network as a discriminator, and then uses the discriminator loss L... D The features of the synthetic and real domains are classified, and the expression is shown in the following formula:

[0091] L D = (z-1)·log(1-D(B) last (I;θ B ),θ D ))-z·log(D(B last (I;θ B ),θ D ))

[0092] In the formula, z is a label indicating whether the input feature comes from a real defocused image or a synthetic defocused image, i.e., whether the input image I is real or synthetic; if the feature is real, then z = 0, otherwise z = 1. B last (I;θ B Returns the last upsampled layer B of image I. last The feature mapping. During training, only the discriminator θ is trained. D The parameters.

[0093] This invention compares the proposed RFAAF-DME network with classic and advanced defocus map estimation networks from recent years. The quality of the images generated by the network is comprehensively evaluated using both objective evaluation metrics and subjective visual effects, verifying the superiority of RFAAF-DME in defocus map estimation. Specifically, this invention selects the classic edge-based defocus map estimation network proposed by Karali et al., the defocus map estimation network based on a convolutional neural network architecture and multiple feature combinations proposed by Park et al., the end-to-end convolutional neural network architecture DME-Net proposed by Lee et al., and DID-ANet proposed by Ma et al., which uses defocus map estimation as an auxiliary task, as comparison networks. By evaluating and comparing the performance of these networks, the superiority of the RFAAF-DME network can be better demonstrated. Figure 5 The quantitative evaluation metrics of different methods on the two datasets are presented separately. A comprehensive analysis of multiple evaluation metrics shows that RFAAF-DME achieved better results overall. The full-reference evaluation metrics show that RFAAF-DME achieved the best performance on both the CUHK and DED datasets. The MAE metric improved by 33.60% and 39.73% compared to the worst results on both datasets, and by 22.84% and 9.06% compared to the second-best result. The high score on the Mean Squared Error (MSE) metric also confirms that the difference between the predicted and actual values ​​of the network in this invention is small, and the model's ability to estimate defocus maps is relatively good.

[0094] Figure 6 and Figure 7This paper compares the visual performance of the proposed RFAAF-DME method with other defocus map estimation algorithms on the CUHK and DED datasets. On the CUHK dataset, the edge-estimation-based Karaali method performs the worst, producing a generally blurry defocus map. Other comparative methods exhibit smudges around objects, while the proposed method shows almost none, indicating stronger robustness in uniform regions. Furthermore, as... Figure 6 As shown in the third row, only the network of this invention effectively represents the blank area between the arm and the body, indicating that the channel attention mechanism introduced in this invention better captures the detailed texture information of the defocused estimation map. On the DED dataset, the method proposed in this invention also demonstrates better performance than other methods, more accurately estimating defocused images and preserving more image detail information. Figure 7 In the second row, the blank space between the bicycle wheel and the railing is also well represented only by the network of this invention.

Claims

1. A defocus map estimation method based on region feature adaptive and attention fusion, characterized in that, The method comprises the steps of: The training step: The defocus estimation network outputs the blur amount of the image to the region feature adaptive network and the cascaded residual block; The training image set includes a training set of synthetic defocus images with labeled defocus maps, a training set of real defocus images without labels, and a training set of real defocus images with labeled binary blur maps; The training set of synthetic defocus images with labeled defocus maps belongs to the synthetic domain, and the training set of real defocus images without labels and the training set of real defocus images with labeled binary blur maps belong to the real domain; The region feature adaptive network classifies the blur amount of the input image from the real domain or the synthetic domain, and outputs the blur amount feature value of the real image or the blur amount of the synthetic image to the cascaded convolution layer residual block; The cascaded residual block outputs the real defocus estimation image or the synthetic defocus estimation image, and inputs the synthetic defocus estimation image classified by the region feature adaptive network into the convolution layer network; The convolution layer network outputs the estimation accuracy of the synthetic defocus estimation image; The defocus estimation network predicts the blur amount of the input image as the training target to confuse the blur amount of the image from the synthetic domain and the real domain; The region feature adaptive network takes the correct classification of the blur amount of the image from the synthetic domain and the real domain as the training target, so as to train the defocus estimation network to minimize the domain difference between the features from the synthetic domain and the real domain; The convolution layer network takes the correct classification of whether each pixel of the synthetic defocus estimation image has the correct blur amount as the training target, so as to train the defocus estimation network to estimate the correct sharpness from the domain transfer feature; The test step: The defocus estimation network is an encoder-decoder structure with added skip connection; N+1 layers of encoders are connected from high to low resolution to form a downsampling structure, and each layer of encoder is a downsampling block; N layers of decoders are connected from low to high resolution to form an upsampling structure, and each layer of decoder is an upsampling block; on the one hand, the feature maps output by the downsampling block and the feature maps of the same resolution output by the upsampling block are spliced by vectors for the first feature fusion, and on the other hand, the feature maps output by each upsampling module are input into the channel attention module for processing, and the feature maps output by the attention mechanism processing are spliced with the skip connection features for the second feature fusion; 2. The method of claim 1, wherein, The feature maps after the second feature fusion are reduced in dimension as the input of the next upsampling block or the output of the final defocus estimation network. The region feature adaptive network is a structure of three layers of convolution layers plus batch normalization layers and then adding a channel attention module.

3. The method of claim 1, wherein, The convolution layer network is a 1x1 convolutional neural network (CNN) structure.

4. The method of claim 1, wherein, ​ 5. The method of claim 1, wherein, In the training step, a loss function L is used g The network parameters θ of the defocus estimation network are trained B and the network parameters θ of the convolutional layer network are trained S , the loss function L g : where |D| is the number of elements in the set D, is the training set of synthetic defocused images with labeled defocused images, where, is the n-th synthetic defocused image I S , y n is the corresponding defocused image; is the training set of real defocused images without labels, R is the n-th real defocused image without labels I is the training set of real defocused images with labeled binary blur maps b n , where, is the n-th real defocused image with labels I B , b n is the binary blur map of the n-th real defocused image I ; L B , L S and L adv are the mean square error loss, the S-shaped cross-entropy loss and the adversarial loss, respectively, and λ S and λ adv are hyperparameters balancing the loss terms.​​​ 6. The method of claim 5, wherein, In the training step, a loss function L is used d Training the network parameters θ of the region feature adaptive network D , the loss function L d : where L D is the discriminator loss, 1 indicates classification as synthetic domain, 0 indicates classification as real domain, λ D is a hyperparameter balancing between L g and L d .

Images