Defocus image estimation method based on regional feature self-adaption and attention fusion

By introducing regional feature adaptation and attention mechanisms in defocus graph estimation, combined with encoder-decoder structure and jump connection, the problem of insufficient real fuzzy data sets is solved, and a more accurate and detailed defocus graph estimation is achieved. , improves the generalization ability of the model and image recovery quality.

CN118968001AActive Publication Date: 2024-11-15UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410964223.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-15
Estimated Expiration
2044-07-18

AI Technical Summary

Technical Problem

The existing defocus graph estimation method has the problem of insufficient number of pictures in real blurred data sets and underutilizing information in low-definition images, resulting in inaccurate defocus estimation graphs output, especially in real scenes.

Method used

Using a defocus map estimation method based on regional feature adaptation and attention fusion, domain differences are narrowed through the synthetic and real domain data in the training set, combined with encoder-decoder structure and jump connection, a channel attention mechanism is introduced to improve the image Restores detail fidelity and designs suitable loss functions to generate realistic output images.

Benefits of technology

A defocus map estimation network with superior performance is built, which can effectively utilize the information in low-definition images, reduce computing resources, improve model generalization capabilities, and generate more accurate and detailed defocus estimation maps.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968001A_ABST
    Figure CN118968001A_ABST
Patent Text Reader

Abstract

The invention provides a defocus image estimation method based on regional feature self-adaption and attention fusion, and the method continues to use a common Encoder-Decoder structure as a basic network and is used for an image restoration task. Secondly, jump connection is added in a basic network, and a channel attention mechanism is introduced in a decoder, so that the decoder can better retain and utilize feature information extracted in an encoder. Meanwhile, the regional feature adaptive network is used for image-to-image regression and is attached to the basic network, the domain difference between the synthetic features and the real features is minimized, the channel attention mechanism is also introduced into the regional feature adaptive network, and the representation capability of the network is further enhanced. And finally, during training, firstly training the regional feature adaptive network as a discriminator, classifying the features of the synthetic domain and the real domain by using the loss of the discriminator, and then training the basic network to minimize the domain difference between the synthetic and real defocused image features.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to computer vision technology, and in particular to a defocus map estimation technology based on deep learning. Background Art

[0002] In videography, the reflection and refraction of light will cause a certain amount of blur when the image is formed. The blurred image is called a defocused image. The amount of blur can be estimated for the defocused image. With the amount of blur, the image can be made clear by adjusting the focal length or angle in the later stage.

[0003] Typical defocus map estimation methods first detect edges from blurred images, then measure the blur around the edges, and finally determine the blur amount in homogeneous regions by interpolation. However, previous edge-driven methods have some limitations. First, since edges in blurred images are usually blurred, their detection results are inaccurate. Second, blur estimation at edges is inherently error-prone because pixels on object boundaries with depth discontinuities usually contain a mixture of different degrees of blur. Third, the instability of edge blur estimation leads to unreliable predictions in homogeneous regions, and the blur amount obtained by relying solely on features defined around image edges is often inaccurate. In other words, the blur amounts estimated for different parts of the object boundary may be unrelated, and then interpolating them into the interior of the homogeneous object will only produce a smooth but less accurate blur estimate. For example, the blur estimate of an object with a single depth may not be constant because the blur amount measured separately for the edge may be different when the edge has different depth discontinuities with nearby objects.

[0004] As deep learning has achieved great results in various computer vision tasks, deep learning-based methods have also been widely used in the defocus map estimation task. Although deep learning methods show great promise in solving image degradation problems, defocus map estimation requires a large amount of accurate training data, which consists of pairs of fully focused and defocused images, which is difficult to collect. The reason is that the two images should be taken under the same exposure and lighting conditions to ensure pixel-level correspondence between the image pairs. However, due to factors such as light changes and camera hardware conditions, ordinary cameras often lead to inconsistent exposure and lighting conditions under different focus settings, which in turn affects the quality of the image pairs. This means that it is difficult to obtain a large number of real-world paired samples to train a defocus map estimation network with good generalization ability. As research develops, many researchers use synthetic blur datasets as a compromise.

[0005] As for the compromise solution, although the image quality of synthetic blur datasets has made great progress thanks to the joint efforts of many scholars, there are still obvious differences in the characteristics of real blur scenes. This leads to the performance of the defocus map estimation network trained using synthetic blur datasets in real defocus blur scenes being seriously affected, and the performance indicators show a clear downward trend. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method that helps to output a realistic defocus estimation map, in view of the problem that the number of real blur data sets is insufficient when estimating the existing defocus map, and the information such as pixels and structures contained in the low-definition images is not fully utilized.

[0007] The technical solution adopted by the present invention to solve the above technical problems is a defocus map estimation method based on regional feature adaptation and attention fusion, comprising:

[0008] Training steps:

[0009] The training image set is input into the defocus estimation network, and the defocus estimation network outputs the blur amount of the image to the regional feature adaptation network and the cascade residual block;

[0010] The training image set includes a training set of synthetic defocus images of defocus images with labels, a training set of real defocus images without labels, and a training set of real defocus images of binary blur images with labels; the training set of synthetic defocus images of defocus images with labels belongs to the synthetic domain, and the training set of real defocus images without labels and the training set of real defocus images of binary blur images with labels belong to the real domain;

[0011] The regional feature adaptive network classifies whether the blur amount of the input image comes from the real domain or the synthetic domain, and outputs the blur amount feature value of the real image or the blur amount of the synthetic image to the residual block of the cascade convolutional layer;

[0012] The cascaded residual block outputs a real defocus estimation image or a synthetic defocus estimation image, and the synthetic defocus estimation image classified by the regional feature adaptation network as a synthetic domain is then input into the convolutional layer network;

[0013] The convolutional layer network outputs the estimated accuracy of the synthetic defocus estimation image;

[0014] The defocus estimation network predicts the blur amount of the input image, and takes the blur amount that confuses images from the synthetic domain and the real domain as the training goal;

[0015] The regional feature adaptation network is trained to correctly classify the blur amount of images from the synthetic domain and the real domain, thereby training the defocus estimation network to minimize the domain difference between the features from the synthetic domain and the real domain;

[0016] The convolutional layer network takes the correct classification of whether each pixel of the synthetic defocus estimation image has the correct blur amount as the training goal, so as to train the defocus estimation network to estimate the correct sharpness from the domain transfer features;

[0017] Test steps:

[0018] The image to be processed is input into the defocus estimation network, and the defocus estimation network outputs the blur amount of the image to complete the defocus map estimation.

[0019] Specifically, the present invention designs an end-to-end trained defocus map estimation network RFAAF-DME: First, the present invention uses the common Encoder-Decoder structure as the basic network for image restoration tasks, wherein the encoder part uses the pre-trained VGG19 network to gradually extract features from the input defocused blurred image, and the decoder part adopts the U-Net structure to fuse the encoder features with the low-level features and expand the feature map output by the encoder to the size of the original input image through upsampling operations. Secondly, jump connections are added to the basic network, and a channel attention mechanism is introduced in the decoder, so that the decoder can better retain and utilize the feature information extracted in the encoder, thereby improving the detail fidelity of image restoration. At the same time, the regional feature adaptive network is used for image-to-image regression, which is attached to the basic network to minimize the domain difference between the synthetic features and the real features, and the channel attention mechanism is also introduced in the regional feature adaptive network to further enhance the representation ability of the network, reduce the network's calculation of unimportant areas, save computing resources and accelerate the network. Finally, during training, the present invention first trains a regional feature adaptation network as a discriminator, uses the discriminator loss to classify the features of the synthetic domain and the real domain, and then trains the base network to minimize the domain difference between the synthetic and real defocused image features, and uses the mean squared error (MSE) and mean absolute error (MAE) of the defocus estimation map loss function to estimate the overall structure of the defocus map and densely predict the amount of blur in the region.

[0020] Based on the above analysis of the deficiencies of existing defocus image estimation algorithms, the present invention focuses on the following points when designing the network: (1) In order to solve the problem of insufficient number of images in the real blur dataset, an improved idea of ​​adding a synthetic blur dataset for training is used, and at the same time, the regional feature adaptive method is considered to narrow the gap between the synthetic training images and the real images, thereby improving the generalization ability of the model. (2) Furthermore, attention is paid to effectively utilizing the pixel, structure and other information contained in the input low-definition image, fully extracting the feature representation in the defocused blur image through the hierarchical feature extraction structure, and reasonably utilizing the position information of the defocused area through the attention mechanism. (3) Furthermore, a loss function is designed that helps to generate realistic output images.

[0021] The beneficial effects of the present invention are:

[0022] (1) A defocused image estimation network based on an encoder-decoder structure was built, and skip connections and channel attention mechanisms were added, so that the network can fully extract and utilize the effective information in the input defocused image.

[0023] (2) In the basic network of defocus map estimation, a regional feature adaptation network that can minimize the features between the real defocused blurred image and the synthetic defocused blurred image is added to construct a defocus map estimation network with good performance.

[0024] (3) A loss function is designed that is conducive to preserving texture and details. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of the backbone network and the skip connection network structure;

[0026] Figure 2 It is the overall network structure of RFAAF-DME;

[0027] Figure 3 It is composed of regional feature adaptive network;

[0028] Figure 4 Schematic diagram of the channel attention structure;

[0029] Figure 5 Quantitative comparison with the classic defocus map estimation network on CUHK and DED datasets;

[0030] Figure 6 To conduct a qualitative comparison with the classic defocus map estimation network on the CUHK dataset;

[0031] Figure 7 To conduct a qualitative comparison with the classic defocus map estimation network on the DED dataset. DETAILED DESCRIPTION

[0032] In the embodiment, the input defocused blurred image is resized to 224×224 and then input into the defocus map estimation network, and finally a defocus estimation image of the same size as the input image is generated.

[0033] The defocus image estimation network includes an initialization layer and a backbone network connected in sequence.

[0034] The initialization layer is used to adjust the input defocused blurred image to obtain a 224×224 feature map.

[0035] The backbone network is an encoder-decoder structure with added skip connections, including N+1 layers of encoders and N layers of decoders; the N+1 layers of encoders are connected from high to low resolution to form a downsampling structure, and each layer of encoder is a downsampling block; the N layers of decoders are connected from low to high resolution to form an upsampling structure, and each layer of decoders is called an upsampling block. The decoder of the last layer restores the output feature map to the input image size of the defocused image estimation network.

[0036] On the one hand, the feature map output by the downsampling block and the feature map of the same resolution output by the upsampling block are fused for the first time through vector concatenation. On the other hand, each upsampling module inputs the output feature map into the attention mechanism for processing, and the feature map output by the attention mechanism is concatenated with the jump connection feature for the second feature fusion; the feature map after the second feature fusion is further reduced in dimension as the input of the next upsampling block or the output of the final defocus map estimation network. In the process of jump connection, a channel attention module is inserted to select features that are helpful for estimating the defocus map.

[0037] like Figure 1 The encoder-decoder structure shown includes a fifth downsampling block, a fourth downsampling block, a third downsampling block, a second downsampling block, a first downsampling block, a first upsampling block, a second upsampling block, a third upsampling block, and a fourth upsampling block connected in sequence. The feature map is gradually reduced in resolution to 14×14 through 5 downsampling blocks, and then gradually increased in resolution through 4 upsampling blocks to generate a feature map of the same size as the initial feature map.

[0038] On the basis of the backbone network, a regional feature adaptation network is added to minimize the features between the real defocused blurred image and the synthetic defocused blurred image using the generative adversarial loss function so that the two regions have the same distribution in terms of extracted features. Finally, after the cascaded residual block, the final result is generated.

[0039] The regional feature adaptive network is a structure of a convolutional layer plus a batch normalization layer cascaded into three layers and then adding a channel attention module, that is, a CNN with three convolutional layers, each of which is followed by a batch normalization layer and a leaky rectified linear unit (Leaky ReLU) activation. Inserting a channel attention module into the regional feature adaptive network further enhances the network's representation ability, reduces the network's calculation of unimportant areas, saves computing resources, and accelerates the network.

[0040] In the process of training the defocus image estimation network, adversarial training is used to stabilize the training.

[0041] The following examples provide detailed descriptions of the various details involved in implementing the scheme of the present invention.

[0042] During the training of the defocus image estimation network B, a regional feature adaptation network D, a cascade structure of multiple residual blocks, and a convolutional layer network S formed by a 1×1 CNN structure are introduced.

[0043] like Figure 2 As shown, during the training process, the real defocused blurred image and the synthetic defocused blurred image are respectively input into the defocus estimation network B, and the defocus estimation network outputs the blur amount of the real image and the blur amount of the synthetic image to the regional feature adaptive network D. The regional feature adaptive network D minimizes the domain difference between the synthetic feature and the real feature and outputs the blur amount feature value of the real image and the blur amount of the synthetic image to the cascade residual block. The cascade residual block outputs the real defocus estimated image and the synthetic defocus estimated image; the synthetic defocus estimated image is then input into the convolutional layer network S, and the convolutional layer network S outputs the accuracy of the synthetic defocus estimated image;

[0044] (1) Build an encoder-decoder defocus map estimation network B based on added skip connections, such as Figure 1 As shown:

[0045] The encoder-decoder structure consists of 5 downsampling blocks and 4 upsampling blocks. First, the defocused image is resized to 224×224 and then input into the backbone network. The resolution is gradually reduced to 14×14 pixels by 5 downsampling blocks, and then gradually increased by 4 upsampling blocks. Finally, a defocused estimated image of the same size as the input image is generated. The image sizes processed by the 5 downsampling blocks are: 224×224×64, 112×112×128, 56×56×256, 28×28×512, 14×14×512; the images processed by the 4 upsampling blocks are: 28×28×512, 56×56×256, 112×112×128, 224×224×64. The first two data of the image size are the image resolution, and the last data is the number of channels.

[0046] Taking into account that different network layers focus on different features, for example, shallow networks focus on texture features, and deep networks focus on global features, both global and texture features are very important for the defocus image estimation task: and due to the downsampling operation, the loss of some edge features is inevitable. Therefore, in order to utilize more and more comprehensive information in the defocused image, the present invention adds skip connections to the backbone network, and vectorizes the feature maps of the same resolution in the encoder and decoder to achieve the effect of feature fusion.

[0047] (2) Add a channel attention module SE based on the above defocus image estimation network B

[0048] In the process of deep learning, the encoder usually extracts a large amount of feature information, but not all extracted features can provide effective information for defocus map estimation. For example, in the input defocused image, there may be some invalid information such as noise and artifacts. If all this information is introduced into the defocus map estimation network, it may cause edge loss and other phenomena in the output image, resulting in poor output image quality. Therefore, the attention mechanism is introduced to optimize this problem. The concept of attention mechanism in computer vision originates from the human brain's perception of the external environment. It can help the neural network focus on key information and ignore unimportant information. It is precisely because of this feature that the attention mechanism can reduce the amount of network computation and improve the efficiency of the network. The present invention adds a channel attention module SE to the feature map extracted by each upsampling block in the decoder to learn which features are helpful in generating a defocus estimation map. At the same time, a channel attention module is also introduced in the regional feature adaptation module to further optimize the performance of the model, such as Figure 2 The working principle of the channel attention module SE is to model the importance of each feature channel and enhance or suppress different channels according to the learning results, thereby improving the model's ability to extract key information.

[0049] like Figure 4 As shown in the figure, the input feature map is processed by a two-branch structure. In the weight branch, there are two key operations: Squeeze and Excitation. In the Squeeze operation, the input feature map (size is h×w×c) is globally averaged pooled and compressed into a 1×1×c feature map with the same number of feature channels, so that the global information of the feature map is obtained. Next, two fully connected layers are used in the Excitation operation. The number of neurons in the first fully connected layer is relatively small, only c / 16, while the number of neurons in the second fully connected layer is the same as the number of input channels. Through the processing of these two fully connected layers, the correlation between channels can be learned and weights can be generated for each feature channel. After that, the Sigmoid activation function is used to fix the learned weight value between 0-1 to obtain the channel weight. Finally, the channel weight is multiplied by the original feature map to change the importance of different channels and achieve adaptive attention to the information between the channels of the feature map.

[0050] (3) Add regional feature adaptive network D

[0051] Domain Feature Adaptation (DFA) is a machine learning method that aims to solve the generalization problem caused by the difference between training data and target domain. In many practical applications, there may only be labeled data from the source domain in the data set, but it is hoped that the model will be applied to the target domain, where there is a certain distribution difference between the source domain and the target domain. The goal of domain feature adaptation is to improve the performance of the model on the target area by utilizing the source domain data without the need for labeled data in the target area. The present invention uses a region generation method and uses it for image-to-image regression to solve the problem of small datasets of real defocused and blurred image pairs. The advantages of the regional feature adaptation method are used to flexibly adjust parameters according to the characteristics and attributes of different regions in real defocused and blurred image pairs and synthetic defocused and blurred image pairs, thereby improving the performance and effect of the defocus map estimation algorithm in images of different defocus degrees and complexities.

[0052] The region-adaptive network of the present invention compares the features of real and synthetic defocused images captured by the blur estimation network. An adversarial training strategy is used on the network during training so that the two regions have the same distribution in terms of extracted features. The region-adaptive network uses a discriminator in the GAN framework, with the ultimate goal of making the captured features of real and synthetic defocused images indistinguishable. Figure 3 As shown, the embodiment designs the regional feature adaptation network as a CNN with four convolutional layers with a convolution kernel of 4×4 and a stride of (2,2). The number of output channels of the four convolutional layers is 64, 128, 256, and 512 respectively, and each convolutional layer is followed by a batch normalization layer BatchNorm and a leaky rectified linear unit (ReLU) activation.

[0053] (4) Train the model and verify its effectiveness through experiments

[0054] The constraints of the defocus image estimation network are mainly divided into the following five points:

[0055] (a) Training objectives

[0056] The ultimate goal of the present invention is to train the blur estimation network B to estimate the blur amount of the real image to obtain the defocus estimation image. To achieve this, three different training sets D are used during training. S , D R and D B The joint training is θ B ,θ D and θ S Parameterized fuzzy estimation network B, regional feature adaptation network D and convolutional layer network S.

[0057] is a training set of synthetic defocused images with labeled defocus maps, where is the nth synthetic defocused image I S ,y n for The corresponding defocus map.

[0058] is a training set of unlabeled real defocused images, represents the nth unlabeled true defocused image I R .

[0059] is a binary fuzzy image b with labels n Real defocused image training set, where b n Each pixel is marked as sharp or blurry. is the nth labeled true defocused image I B , b n is the nth real defocused image The binary fuzzy image.

[0060] The real defocused blurred images during training come from D R and D B , synthetic defocused blurred image from D S .

[0061] Given a training dataset, according to the common practice of adversarial training, the loss function L is used g Training θ B and θ S , using the loss function L d Training θ D .

[0062] Loss function L g :

[0063]

[0064] Where |D| represents the number of elements in the set D. B , L S and L adv They are mean square error loss, S-type cross entropy loss and adversarial loss. S and λ adv is a hyperparameter of the balance loss term.

[0065] Loss function L d :

[0066]

[0067] Where L D is the discriminator loss, λ D YesLg and L d A hyperparameter that balances the two, 1 means classification as a synthetic domain, and 0 means classification as a real domain.

[0068] During training, networks D and S influence network B differently depending on the domain of the input. In the case of synthetic blurred images with labeled defocus maps, S ,y>∈D S , using the blur graph loss L which measures the mean squared error (MSE) B Directly minimize the defocus map y and the defocus estimation image B(I S The defocus estimation image output by network B is the predicted defocus map.

[0069] A true defocused image with a binary blur map, B ,b>∈D B , used to calibrate the sharpness measurement from the domain transfer feature. Under the supervision of b, the sigmoid cross entropy loss function L is used for sharpness calibration S Guide network S to estimate the defocus image B(I B ) has the correct amount of blur and finally calibrates network B to estimate the correct sharpness from the domain transferred features.

[0070] Finally, I S ∈D S ,I B ∈D B and I R ∈D R Together they are used to minimize the domain difference between the features extracted from the synthetic and real defocused images. Image I S From the synthetic domain, I B and I R From the real domain. The present invention minimizes the discriminator loss L in an adversarial manner D and adversarial loss L adv , where network D is trained to correctly classify feature domains from different inputs, while network B is trained to confuse D.

[0071] The following is a detailed introduction to the mean square error loss function, adversarial loss function, cross entropy loss function, and discriminator loss function designed in the overall training loss function.

[0072] (b) Mean square error loss function

[0073] The mean square error (MSE) loss function L is used during the training of the defocus image estimation network B. B To estimate the overall structure of the defocus map and densely predict the amount of blur in the region. Given a synthetic defocused image I of size W×H S , L​​B The definition is shown in the following formula:

[0074]

[0075] Among them, B(I S θ B ) i,j is the defocus estimation network with learned parameters θ B The predicted I at the pixel (i, j) S Blur amount, y i,j is the value at pixel (i, j) in the defocus map.

[0076] L aux is the proportional auxiliary loss, defined as follows:

[0077]

[0078] Among them B l (I S θ B ,θ aux )=A l (B l (I S θ B );θ aux ) is the output of the network B upsampled to level l, parameterized as θ aux A small auxiliary network A l Converted to a defocused map. Each auxiliary network A l It consists of two convolutional layers, where the number of kernels in the first layer varies with the number of layers l. aux is a balance parameter. l ×H l is the size of the l-level defocus map. l,i,j is adjusted to W l ×H l Defocus map of size. L B is the number of upsampling layers in B.

[0079] (c) Adversarial Loss Function

[0080] Next, the present invention trains the defocus map estimation network to minimize the domain difference between the synthetic and real defocus image features. R , define the domain adaptive adversarial loss L adv As shown in the following formula:

[0081] L adv = -log(D(B last (I R θ B ),θ D ))

[0082] Fixed parameter θ D , only train θ B The main goal here is to train the defocus map estimation network so that it treats real and synthetic defocused images as coming from the same domain. As the region feature adaptation network becomes more powerful as a region classifier, the defocus map estimation network needs to generate more features that cannot distinguish between real and synthetic regions to minimize the inter-regional differences in feature extraction.

[0083] (d) Cross Entropy Loss Function

[0084] The regional feature adaptation network D focuses on modulating the overall distribution of features extracted from real and synthetic defocused images, rather than specifically focusing on aligning the blur amount of corresponding features between the two regions. In other words, the blur estimation network B cannot be directly applied to real defocused images because the blur amount it obtains when learning synthetic defocused images cannot be easily applied to real images. Therefore, it is necessary to calibrate the blur estimates of the two domains during training. To address this issue, additional information is provided to the features extracted from these images by correlating the real defocused images output by network B with the blur information in an available blur detection dataset. In this process, each pixel is labeled as sharp or blurry.

[0085] For a given synthetic defocused image in the dataset, we train a network S to classify the output of network B to check whether the estimated blur is correct. The prediction is considered correct only if the pixels estimated to be sharp are located in the sharp area of ​​the input image. We construct a network S with 1×1 convolutional layers, and use batch normalization and leaky ReLU layers after each convolutional layer to force network B to estimate blur densely.

[0086] Optimization is performed using sigmoid cross entropy loss:

[0087]

[0088] where b i,j is the value of the binary blur map b at pixel (i, j), which is a true binary blur map. A 1×1 kernel is used during training to keep the receptive field size between networks B and S the same. Otherwise, as the receptive field of S becomes larger, the gradient transferred from S to B will propagate to an area larger than the receptive field of B.

[0089] (e) Discriminator loss function

[0090] The present invention first trains a regional feature adaptive network as a discriminator, and uses the discriminator loss L D The features of the synthetic domain and the real domain are classified, and the expression is shown in the following formula:

[0091] L D =(z-1)·log(1-D(B last (I; θ B ),θ D ))-z·log(D(B last (I; θ B ),θ D ))

[0092] Where z is a label indicating whether the input feature comes from a real defocused image or a synthetic defocused image, that is, whether the input image I is real or synthetic; if the feature is real, z = 0, otherwise z = 1. last (I; θ B ) returns the last upsampling layer B of image I last During the training process, only the discriminator θ is trained D Parameters.

[0093] The present invention selects classic and advanced defocus image estimation networks in recent years to compare with the RFAAF-DME network proposed in the present invention, comprehensively evaluates the quality of network-generated images through objective evaluation indicators and subjective visual effects, and verifies the advancement of RFAAF-DME in defocus image estimation. Specifically, the present invention selects the classic edge-based defocus image estimation network proposed by Karaali et al., the defocus image estimation network based on convolutional neural network architecture and multiple feature combinations proposed by Park et al., and the end-to-end convolutional neural network architecture DME-Net proposed by Lee et al. and the DID-ANet proposed by Ma et al. with defocus image estimation as an auxiliary task as comparison networks. By evaluating and comparing the performance of these networks, the superiority of the RFAAF-DME network can be better demonstrated. Figure 5 The quantitative evaluation index results of different methods on the two datasets are shown respectively. Combining multiple evaluation indicators, it can be concluded that RFAAF-DME has achieved better results overall. From the full reference evaluation index results, it can be seen that RFAAF-DME has achieved the best results on both the CUHK dataset and the DED dataset. The MAE index on the two datasets has increased by 33.60% and 39.73% respectively compared with the worst results, and by 22.84% and 9.06% respectively compared with the second place. The scores obtained on the mean square error MSE index also confirm that the difference between the predicted value and the true value of the network of the present invention is small, and the model's ability to estimate the defocus map is relatively good.

[0094] Figure 6 and Figure 7The figure is a visual comparison of the RFAAF-DME method proposed in the present invention and other defocus map estimation algorithms on the CUHK dataset and the DED dataset. On the CUHK dataset, the edge estimation-based Karaali method performs the worst, and the output defocus estimation map is blurred as a whole. Other comparison methods have smudges around the object, while the method proposed in the present invention has almost no smudges, which shows that the method of the present invention has stronger robustness in uniform areas. In addition, Figure 6 As shown in the third row, only the network of the present invention can better represent the blank part between the arm and the body, which indicates that the channel attention mechanism introduced by the network of the present invention can better obtain the detailed texture information of the defocus estimation image. On the DED dataset, the method proposed by the present invention also shows better performance than other methods, and can estimate the defocused image more accurately and retain more image detail information. Figure 7 In the second row, the blank area between the bicycle wheel and the railing is also well represented only by the network of the present invention.

Claims

1. A defocus map estimation method based on regional feature adaptation and attention fusion, characterized in that: Includes steps: Training steps: The training image set is input into the defocus estimation network, and the defocus estimation network outputs the blur amount of the image to the regional feature adaptation network and the cascade residual block; The training image set includes a training set of synthetic defocused images with labeled defocus maps, a training set of real defocused images without labels, and a training set of real defocused images with labeled binary blur maps; The training set of synthetic defocused images with labeled defocus maps belongs to the synthetic domain, and the training set of real defocused images without labels and the training set of real defocused images with labeled binary blur maps belong to the real domain; The regional feature adaptive network classifies whether the blur amount of the input image comes from the real domain or the synthetic domain, and outputs the blur amount feature value of the real image or the blur amount of the synthetic image to the residual block of the cascade convolutional layer; The cascaded residual block outputs a real defocus estimation image or a synthetic defocus estimation image, and the synthetic defocus estimation image classified by the regional feature adaptation network as a synthetic domain is input into the convolutional layer network; The convolutional layer network outputs the estimated accuracy of the synthetic defocus estimation image; The defocus estimation network predicts the blur amount of the input image, and takes the blur amount that confuses images from the synthetic domain and the real domain as the training goal; The regional feature adaptation network is trained to correctly classify the blur amount of images from the synthetic domain and the real domain, thereby training the defocus estimation network to minimize the domain difference between the features from the synthetic domain and the real domain; The convolutional layer network takes the correct classification of whether each pixel of the synthetic defocus estimation image has the correct blur amount as the training goal, so as to train the defocus estimation network to estimate the correct sharpness from the domain transfer features; Test steps: The image to be processed is input into the defocus estimation network, and the defocus estimation network outputs the blur amount of the image to complete the defocus map estimation.

2. The method according to claim 1, characterized in that: The defocus estimation network is an encoder-decoder structure with skip connections added; N+1 layers of encoders are connected from high to low resolution to form a downsampling structure, and each layer of encoders is a downsampling block; N layers of decoders are connected from low to high resolution to form an upsampling structure, and each layer of decoders is called an upsampling block; on the one hand, the feature map output by the downsampling block and the feature map of the same resolution output by the upsampling block are vector-concatenated for the first feature fusion, and on the other hand, each upsampling module inputs the output feature map into the channel attention module for processing, and the feature map output by the attention mechanism is concatenated with the skip connection feature for the second feature fusion; The feature map after the second feature fusion is further reduced in dimension as the input of the next upsampling block or the output of the final defocus map estimation network.

3. The method according to claim 1, characterized in that: The regional feature adaptation network is a structure consisting of three cascaded layers of convolutional layers and batch normalization layers, and then a channel attention module is added.

4. The method according to claim 1, characterized in that: The convolutional layer network is a 1×1 convolutional neural network CNN structure.

5. The method according to claim 1, characterized in that: In the training step, the loss function L is used g Train the network parameters θ of the defocus estimation network B and the network parameters θ of the convolutional layer network S , the loss function L g : Among them, |D| represents the number of elements in the set D. is a training set of synthetic defocused images with labeled defocus maps, where is the nth synthetic defocused image I S ,y n for The corresponding defocus map; is a training set of unlabeled real defocused images, represents the nth unlabeled true defocused image I R ; is a binary fuzzy image b with labels n Real defocused image The training set, is the nth labeled true defocused image I B , b n is the nth real defocused image Binary fuzzy graph of L B , L S and L adv They are mean square error loss, S-type cross entropy loss and adversarial loss, λ S and λ adv are all hyperparameters of the balanced loss term.

6. The method according to claim 5, characterized in that In the training step, the loss function L is used d The network parameters θ of the training regional feature adaptation network D , the loss function L d : Among them, L D is the discriminator loss, 1 indicates classification as synthetic domain, 0 indicates classification as real domain, λ D YesL g and L d The hyperparameters that are balanced between .

Citation Information

Patent Citations

  • Defocusing fuzzy detection method based on attention mechanism

    CN111563513A

  • Feature domain single image defogging method based on domain transformation

    CN112907478A

  • Defocused image deblurring method and device based on priori knowledge and medium

    CN115456914A

  • Full-pixel dual-core image defocus blurring removal method based on language driving

    CN117934329A