Depth feature enhanced single image depth-of-field expansion method

Through the single-image depth of field expansion method with depth feature enhancement, the multi-scale feature enhancement and image feature optimization fusion structure is used, combined with the dynamic sliding multi-head attention module and the depth enhancement feedforward network, the accuracy and applicability of single-image depth of field recovery are solved. The problem of this achieves efficient depth of field expansion and natural texture information recovery.

CN119068035APending Publication Date: 2024-12-03ZHENGZHOU UNIVERSITY OF LIGHT INDUSTRY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411114293.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art lacks efficient recovery methods in the expansion of the depth of field in single image, and the insufficient data set, resulting in insufficient accuracy and applicability of the depth of field recovery, and defuzzing technology will lead to distortion and transition smoothing.

Method used

Using a single image depth of field expansion method with depth feature enhancement, by building a model including encoder, Bottleneck structure and decoder, using multi-scale feature enhancement structure and image feature optimization of fusion structure, combining dynamic sliding multi-head attention module and depth enhancement feedforward Network, design mixed loss functions for training to achieve effective depth of field expansion of single-focus images.

Benefits of technology

It improves the speed of image acquisition and fusion, reduces the difficulty of data processing, reduces image detail distortion and transition smoothing, improves the visual effect, and makes the depth of field expansion effect of single-focus image fusion algorithm close to the multi-focus image fusion algorithm.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068035A_ABST
    Figure CN119068035A_ABST
Patent Text Reader

Abstract

The invention provides a depth feature enhanced single-image depth-of-field expansion method, which comprises the following steps of: collecting and establishing a single-image depth-of-field expansion data set, and selecting a training set, a verification set and a test set from the single-image depth-of-field expansion data set; a single image depth-of-field expansion model is built, the model comprises an encoder, a Bottleneck structure and a decoder which are connected in sequence, and the encoder, the Bottleneck structure and the decoder each comprise a multi-scale feature enhancement structure and an image feature optimization fusion structure which are connected in sequence; designing a mixed loss function, and respectively training and verifying the single image depth-of-field expansion model by using the training set and the verification set; and performing depth-of-field expansion on the single-focus image provided by the test set by using the finally trained single-image depth-of-field expansion model. According to the method, the processing efficiency can be improved for the problems of low image acquisition, image alignment and fusion speed, insufficient robustness and the like, and the data processing difficulty is greatly reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of optics and image processing, and in particular to a single image depth of field extension method with depth feature enhancement. Background Art

[0002] Depth of field is defined as the range within which a camera lens or other imaging device maintains sharp focus within a certain distance in front and behind it. Due to the limited depth of field of an optical lens, it is difficult for an imaging device to keep all objects or areas in a scene sharp and in focus. This limitation causes blurring in areas outside the focus range of the depth of field. There is often a lot of valuable information in the blurred areas outside the boundaries of the focus area. In addition, there is a lot of potential information in the blurred areas outside the focus area, which is very important for many optical research and computer vision tasks, including microscopic imaging, security monitoring, and target detection.

[0003] Hardware methods for extending the depth of field include the use of special lens combinations, optical coatings, and multi-camera systems. However, as the depth of field range increases, the complexity and cost of these hardware optical systems also increase significantly. Importantly, no matter how the optical system on the hardware is improved and perfected, it is impossible to obtain fully focused image data.

[0004] With the development of deep learning in optics and computer vision, the depth of field extension technology based on deep learning is divided into multi-image depth of field extension (multi-focus fusion) and single-image depth of field extension. Multi-image depth of field extension is to generate a fully focused image by merging data from multiple partially focused images of the same scene. However, multi-image depth of field extension usually requires aligned and calibrated images from different focus areas, which brings problems such as image acquisition, image alignment, slow fusion speed and insufficient robustness. Due to the inherent challenges of multi-image depth of field extension, researchers have tried to use deblurring techniques to solve the depth of field extension problem. These techniques usually estimate the defocus map and apply non-blind deconvolution, which is a process involving estimating the blur kernel.

[0005] For example, the invention patent with application number CN202211390819.4 discloses the construction of a deep learning network model for single image defocus image blur, which includes an encoder and a decoder; a prior knowledge-driven feature extraction module is added to the deep learning network model as an encoder for feature extraction; the light field generation dataset LFDOF is used as the first training dataset to train the deep learning network model, and the DPDD dataset is used as the second training dataset to fine-tune the deep learning network model; the trained deep learning network model restores a clear image from the defocused image. The advantage of this invention is that a prior knowledge-driven feature extraction module is introduced into the network model, which uses shared convolution kernels to extract public information and private convolution kernels to extract modality-specific information to better handle spatially varying defocus blur.

[0006] However, current defocus map algorithms lack sufficient accuracy, reduce data quality, and require defocus map estimation before processing, which limits their applicability. In addition, there is currently a lack of professional datasets for single image depth of field extension, which has caused certain limitations in the study of single image extension. Therefore, it is very important to study a method that can efficiently restore the depth of field of single focus blurred images and a professional dataset for single image depth of field extension. Summary of the invention

[0007] In order to solve the technical problem that the existing depth of field extension technology requires complex defocus maps as an auxiliary or deblurring technology, which causes distortion or smooth transition of the original focus and blur positions, the present invention proposes a single image depth of field extension method with depth feature enhancement, which can achieve effective and convenient depth of field restoration for a single focused image and achieve good visual effects.

[0008] In order to achieve the above object, the technical solution of the present invention is achieved as follows:

[0009] A single image depth of field extension method with depth feature enhancement includes the following steps:

[0010] S1: Collect and establish a single image depth of field extension dataset, and select a training set, validation set, and test set from the single image depth of field extension dataset;

[0011] S2: Build a single image depth of field extension model, including an encoder, a Bottleneck structure and a decoder connected in sequence. The encoder, the Bottleneck structure and the decoder all contain a multi-scale feature enhancement structure and an image feature optimization fusion structure connected in sequence.

[0012] S3: Design a hybrid loss function and use the training set and validation set to train a single image depth of field extension model to obtain a trained depth of field extension model;

[0013] S4: Use the final trained single image depth of field extension model to extend the depth of field of the single focus image provided by the test set to obtain an image with extended depth of field.

[0014] In step S1, pairs of single-focus images and all-focus images corresponding to the single-focus images in the single-image depth of field extension dataset are used as training sets and verification sets, and single-focus images without all-focus image comparison in the single-image depth of field extension dataset and captured real single-focus images are used as test sets.

[0015] In the single image depth of field extension model described in step S2, each layer of the encoder and each layer of the decoder respectively includes a residual structure and a multi-scale feature enhancement structure I and an image feature optimization fusion structure I connected in sequence, and the Bottleneck structure includes a multi-scale feature enhancement structure II and an image feature optimization fusion structure II connected in sequence; the image feature optimization fusion structures I and II both include a dynamic sliding multi-head attention module and a deep enhanced feedforward network connected in sequence using a sliding window operation.

[0016] The method for training a single image depth of field extension model described in step S3 is as follows: a training set is input, a single focus image I in the input training set is subjected to a convolution operation to obtain a feature map B1, the feature map B1 is sequentially subjected to a multi-scale feature enhancement structure and an image feature optimization fusion structure of the first layer of the encoder to obtain a high-level feature G1 after the first layer of the encoder enhancement fusion, the high-level feature G1 and the feature map B1 are subjected to a residual connection to obtain deep feature information S1, the deep feature information S1 is downsampled, and sequentially passes through the same encoder layer of M-1 layers to obtain all feature information SM after the single focus image enhancement fusion, the feature information SM is downsampled, and then passes through the multi-scale feature in the Bottleneck structure. The feature enhancement structure and the image feature optimization fusion structure are used to obtain the high-level feature BL after enhancement and fusion in the Bottleneck structure. The high-level feature BL is connected and combined with the feature information SM from the jump connection after upsampling, and then the feature map JM is obtained after the convolution operation. The feature map JM is sequentially passed through the multi-scale feature enhancement structure and the image feature optimization fusion structure, and then the residual link is performed with the feature map JM. Then, the predicted full-focus map YM after decoding by the Mth layer of the decoder is obtained after upsampling. The predicted full-focus map YM passes through the same decoding layer of the M-1 layer in turn, and finally after the convolution operation, the element-by-element addition operation is performed with the single-focus image I in the input training set to obtain the final predicted full-focus map x;

[0017] A hybrid loss function is used to calculate the loss of the single image depth of field extension model and adjust the model network structure. The validation set is used to evaluate the performance of the model. The hyperparameters of the model are dynamically adjusted according to the performance on the validation set. The training is repeated until the best hyperparameter combination is found to obtain the final trained single image depth of field extension model.

[0018] The calculation process in the multi-scale feature enhancement structures I and II is as follows: the input feature map Fb of each multi-scale feature enhancement structure passes through three pooling layers with different step sizes to extract the global prior information of each layer of the encoder or decoder, and the global prior information extracted by each pooling layer is subjected to a convolution operation, and then further enhanced and optimized by channel attention calculation and spatial attention calculation to obtain enhanced and optimized global prior information. Finally, these enhanced and optimized global prior information are aggregated, passed through a convolution layer, and then combined with the input feature map Fb through a residual to obtain an enhanced global feature F, and the enhanced global feature F is sent to the image feature optimization fusion structure.

[0019] The method for calculating the channel attention is as follows: each input feature map Fa is subjected to global maximum pooling and global average pooling to generate new feature vectors and, then these two feature vectors are passed through a shared multi-layer perceptron MLP to generate two new feature vectors and, then the two feature vectors are added together, and a channel attention map M is generated through a sigmoid activation function. C =sigmoid(M max +M avg ), and finally multiply the channel attention map by the original feature map element by element to obtain the final output feature map F out =M C ⊙Fa;

[0020] The method for calculating the spatial attention is: output feature map F out After maximum pooling and average pooling, a new feature map F is generated max1 and F avg1 Next, the two feature maps are concatenated to obtain the feature map F concat , and then perform a convolution operation on the concatenated feature map to generate a spatial attention map M S =sigmoid(Conv(F concat )), and finally combine the spatial attention map with the output feature map F out Perform element-by-element multiplication to obtain the final output feature map F out1 =M S ⊙F out .

[0021] The calculation process in the image feature optimization fusion structure I and II is as follows: the enhanced global feature F is passed through a dynamic sliding multi-head attention module and a deep enhanced feedforward network to obtain enhanced and fused high-level features.

[0022] The calculation process in the dynamic sliding multi-head attention module is: use two deep convolutions to calculate the enhanced global feature F to obtain the query key Sum Where W i (·) is a 1×1 depthwise convolution, It is a 3×3 depth convolution, which divides the enhanced global feature F. For each divided feature window i, the divided query Q is used i , key K i Sum value V i The local attention map of the corresponding size is generated by the softmax activation function; the calculation formula of the local attention map is:

[0023]

[0024] where d K is the dimension of the key K, and then the local attention map Attention(Q i ,K i ,V i ) to obtain the attention output map of the complete sequence; the attention output map is residually connected with the enhanced global feature F, and then normalized to obtain the final attention output map;

[0025] The calculation process in the deep enhanced feedforward network is as follows: the final attention output map passes through a feedforward network containing three layers of deep convolution, the feedforward network output is residually connected with the feedforward network input, and then normalized to obtain enhanced and fused high-level features.

[0026] The method for calculating the loss of a single image depth of field extension model using a hybrid loss function is as follows: using a multi-scale structural similarity loss function to perform structural similarity loss calculation on each layer of the decoder of the single image depth of field extension model, using a mean absolute error loss function to perform mean absolute error loss calculation on the final predicted all-focus image x and the real all-focus image y in the training set, mixing the multi-scale structural similarity loss function and the mean absolute error loss function to obtain a hybrid loss function and performing hybrid loss calculation on the single image depth of field extension model.

[0027] The method for calculating the structural similarity loss is:

[0028] L MS-SSIM =1-MS-SSIM(x,y)

[0029]

[0030] Where M represents the number of decoder layers, MS-SSIM(x,y) represents the multi-scale structural similarity, and l j(x, y) is the brightness similarity between the predicted all-focus image and the corresponding all-focus image at the jth layer of the decoder, c j (x, y) is the contrast similarity between the predicted all-focus image and the corresponding all-focus image at the jth layer of the decoder, s j (x, y) is the structural similarity between the predicted image of the jth layer of the decoder and the corresponding all-focus image;

[0031] The method for calculating the mean absolute error loss is:

[0032]

[0033] Where y is the real all-focus image in the input training set corresponding to the final predicted all-focus image x, N represents the total number of pixels in the image, and x k is the kth pixel in the final predicted full focus image x of the single image depth of field extension model, y k is the pixel x in the true all-focus image y k The corresponding k-th pixel;

[0034] The method for calculating the mixed loss is: Mix =λL MS-SSIM +(1-λ)L1

[0035] Among them, L MS-SSIM is the multi-scale structural similarity loss function, L1 is the mean absolute error loss function, and λ is the parameter used to balance the two loss terms.

[0036] Beneficial effects of the present invention:

[0037] Compared with multi-image depth of field extension technology, the present invention can improve processing efficiency and significantly reduce the difficulty of data processing in terms of image acquisition, image alignment, slow fusion speed and insufficient robustness. Compared with the deblurring technology in single-frame depth of field extension, the present invention can reduce the distortion of image details and smooth transition in the original focus and blurred areas, and achieve better visual effects. The present invention can achieve a more effective depth of field extension for single-focus images, restore texture information and tones naturally, and is not prone to artifacts, distortion and other phenomena, and can be comparable to the structure of multi-focus image fusion algorithms to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0040] Figure 2 It is a schematic diagram of the network structure of the present invention.

[0041] Figure 3 Sample diagrams of the training set and test set TEST(A) and test set TEST(B) proposed by the method of the present invention.

[0042] Figure 4 The present invention is based on Figure 3 Comparison chart of the test set TEST(B) with the results of existing methods, where (a) is a single focus image, (b) is the depth of field extension image obtained by the model DRBNet, (c) is the depth of field extension image obtained by the model MIMO-UNet, (d) is the depth of field extension image obtained by the model MPRNet, (e) is the depth of field extension image obtained by the model Uformer, and (f) is the depth of field extension image of the method of the present invention.

[0043] Figure 5 This is a comparison chart of the results of the present invention based on the public data sets DPDD and RealDOF and the existing methods, where (a) is a single focus image, (b) is the depth of field extension image obtained by the model MTRNN, (c) is the depth of field extension image obtained by the model MPRNet, (d) is the depth of field extension image obtained by the model MIMO-UNet, (e) is the depth of field extension image obtained by the model DRBNet, (f) is the depth of field extension image obtained by the model Uformer, (g) is the depth of field extension image of the method of the present invention, and (h) is the full-focus image corresponding to (a).

[0044] Figure 6 This is an example diagram comparing the depth of field extension and multi-focus image fusion results of the present invention based on 11 groups of collected uncalibrated images. The input is 13 uncalibrated images, CWT is the fused image obtained by the model CWT, DSIFT is the fused image obtained by the model DSIFT, GFDF is the fused image obtained by the model GFDF, SESF is the fused image obtained by the model SESF, and SDENet is the depth of field extension image of the method of the present invention.

[0045] Figure 7 This is an example diagram comparing the depth of field extension and multi-focus image fusion results based on the collected 9 groups of calibrated and aligned images. The input is 8 calibrated and aligned images, CWT is the fused image obtained by the model CWT, DSIFT is the fused image obtained by the model DSIFT, GFDF is the fused image obtained by the model GFDF, SESF is the fused image obtained by the model SESF, and SDENet is the depth of field extension image of the method of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] like Figure 1 As shown, a single image depth of field extension method with deep feature enhancement is described, and the specific steps are as follows:

[0048] S1: Collect and establish a single image depth of field extension dataset, and select a training set, validation set, and test set from the single image depth of field extension dataset.

[0049] Paired single-focus images and their corresponding all-focus images in the single image depth of field extension dataset are used as training sets and validation sets, and single-focus images without all-focus image controls in the single image depth of field extension dataset and captured real single-focus images are used as test sets.

[0050] Since the prior art lacks a single image depth of field extension dataset, the present invention selects those images that are not globally defocused and have prominent and clear focus areas from the DEDD dataset, DPDD dataset, and LFDOF dataset, respectively, and the images outside the focus area are blurred due to the limitations of the imaging device, and are randomly divided into training sets and validation sets, with a total of 1,772 pairs. The all-focus images are taken by dual-pixel sensors and image fusion technology, and light field cameras, respectively. There are also two test sets. The test set TEST (A) contains 387 single focus images selected from the DEDD test set, DPDD test set, and LFDOF test set. The test set TEST (B) contains a total of 100 single focus images taken by mobile phones such as Xiaomi, Honor, Apple, and Vivo. Image examples of the single image depth of field extension dataset are shown in the following figure. Figure 3 shown.

[0051] S2: Build a single image depth of field extension model, such as Figure 2 As shown, it includes an encoder, a Bottleneck structure and a decoder connected in sequence, each layer of the encoder and each layer of the decoder respectively includes a residual structure and a multi-scale feature enhancement structure I and an image feature optimization fusion structure I connected in sequence, the Bottleneck structure includes a multi-scale feature enhancement structure II and an image feature optimization fusion structure II connected in sequence, and the image feature optimization fusion structure is a Transformer structure, including a dynamic sliding multi-head attention module using a sliding window operation and a deep enhanced feedforward network connected in sequence;

[0052] S3: Design a hybrid loss function, use the training set and the validation set to train the single image depth of field extension model, and obtain the trained depth of field extension model.

[0053] The method of using the training set to train a single image depth of field extension model is:

[0054] like Figure 2 As shown, the input training set, the single focus image in the input training set After the 3×3 convolution operation, the feature map B1 is obtained. The feature map B1 is sequentially passed through the multi-scale feature enhancement structure and image feature optimization fusion structure of the first layer of the encoder to obtain the high-level feature G1 after the first layer of encoder enhancement fusion. The high-level feature G1 and the feature map B1 are connected by residual connection to obtain the deep feature information S1. After the deep feature information S1 is down-sampled by halving the feature map size and doubling the number of channels, it is sequentially passed through 3 layers of the same encoder layer to obtain all the feature information S4 after single focus image enhancement fusion. After the feature information Sm is down-sampled by halving the feature map size and doubling the number of channels, it is passed through the multi-scale feature enhancement structure and image feature optimization fusion structure in the Bottleneck structure to obtain Bottle. The high-level feature BL after enhancement and fusion in the neck structure is connected and combined with the feature information S4 from the jump connection after upsampling with doubling the feature map size and halving the number of channels, and then undergoes a 1×1 convolution operation to obtain the feature map J4. The feature map J4 is successively passed through the multi-scale feature enhancement structure and the image feature optimization fusion structure for residual connection with the feature map J4, and then is upsampled with doubling the feature map size and halving the number of channels to obtain the predicted full-focus map Y4 after decoding by the 4th layer of the decoder. The predicted full-focus map Y passes through 3 identical decoding layers in sequence, and finally undergoes a 3×3 convolution operation and performs an element-wise addition operation with the single-focus image I in the input training set to obtain the final predicted full-focus map x.

[0055] The calculation process in the multi-scale feature enhancement structures I and II is as follows: the input feature map Fb of each multi-scale feature enhancement structure passes through three pooling layers with different step sizes. In this embodiment, the step sizes are 1, 2, and 4, and the global prior information of each layer of the encoder or decoder is extracted. The global prior information extracted by each pooling layer is subjected to a convolution operation, and then further enhanced and optimized by spatial attention (SAM) calculation and channel attention (CAM) calculation to obtain enhanced and optimized global prior information. Finally, these enhanced and optimized global prior information are aggregated, passed through a convolution layer, and then combined with the feature map Fb through a residual to obtain an enhanced global feature F, and the enhanced global feature F is sent to the image feature optimization fusion structure.

[0056] The channel attention calculation method is as follows: each channel attention input feature map F1 is subjected to global maximum pooling and global average pooling to generate a new feature vector F max and F avg Next, these two feature vectors are passed through a shared multi-layer perceptron MLP to generate two new feature vectors M max and M avg , then add the two feature vectors and generate the channel attention map M through the sigmoid activation function C =sigmoid(M max +M avg ), and finally multiply the channel attention map by the original feature map element by element to obtain the final output feature map F out =M C ⊙F1.

[0057] The spatial attention calculation method is as follows: Input feature map F out After maximum pooling and average pooling, a new feature map F is generated max1 and F avg1 Next, the two feature maps are concatenated to obtain the feature map F concat , and then perform a convolution operation on the concatenated feature map to generate a spatial attention map M S =sigmoid(Conv(F concat )), and finally combine the spatial attention map with the output feature map F out Perform element-by-element multiplication to obtain the final output feature map F out1 =M S ⊙F out .

[0058] The calculation process in the image feature optimization fusion structure I and II is as follows: the enhanced global feature F is passed through a dynamic sliding multi-head attention module and a deep enhanced feedforward network to obtain enhanced fused high-level features;

[0059] In the dynamic sliding multi-head attention module, 1×1 and 3×3 deep convolutions are used instead of linear layers to calculate the enhanced global feature F to obtain the query key Sum Where W i (·) is a 1×1 depthwise convolution, It is a 3×3 deep convolution, which divides the enhanced global feature F of the input multi-head attention mechanism. For each divided feature window i, the divided query Q is used i , key K i Sum value V iThe local attention map of the corresponding size is generated through the dot product interaction through the softmax activation function; the calculation formula of the local attention map is:

[0060]

[0061] where d K is the dimension of key K;

[0062] Then, the local attention map Attention (Q i ,K i ,V i ) are combined to obtain the attention output map of the complete sequence; the attention output map obtained by the multi-head attention mechanism is residually connected with the input feature map of the multi-head attention mechanism (the enhanced global feature F), and then normalized to obtain the final attention output map.

[0063] In the deep enhanced feedforward network, the final attention output map passes through a feedforward network containing three layers of deep convolution, namely 1×1 deep convolution, 3×3 deep convolution and 1×1 deep convolution. The first layer of 1×1 deep convolution is used to increase the feature channel and pave the way for 3×3 deep convolution to capture local information. The third layer of 1×1 deep convolution is used to restore the deep feature map to the original dimension. After the feedforward network output is residually connected with the feedforward network input, normalization is performed to obtain enhanced and fused high-level features.

[0064] A hybrid loss function is designed to measure the difference between the all-focus image predicted by the single-image depth of field extension model and the real all-focus image in the input training set. Hyperparameters such as the learning rate are dynamically adjusted according to the performance on the validation set to optimize the training process.

[0065] The multi-scale structural similarity loss function is used to calculate the structural similarity loss for each layer of the decoder of the single image depth of field extension model. The structural similarity (SSIM) value of the predicted all-focus image and the corresponding all-focus image of each layer of the decoder is calculated. The structural similarity (SSIM) values ​​corresponding to the four layers of the decoder are combined to obtain the multi-scale structural similarity loss.

[0066] The multi-scale structural similarity loss function L MS-SSIM The calculation formula is:

[0067] L MS-SSIM =1-MS-SSIM(x,y)

[0068]

[0069] Where M = 4 represents the number of decoder layers, MS-SSIM(x,y) represents multi-scale structural similarity, and lj (x, y) is the brightness similarity between the predicted all-focus image and the corresponding all-focus image at the jth layer of the decoder, c j (x, y) is the contrast similarity between the predicted all-focus image and the corresponding all-focus image at the jth layer of the decoder, s j (x, y) is the structural similarity between the predicted all-focus image and the corresponding all-focus image at the jth layer of the decoder.

[0070] In the single image depth of field extension model, the mean absolute value error loss function is used to calculate the mean absolute value error loss of the final predicted all-focus image x and the real all-focus image y in the training set.

[0071] The calculation formula of the mean absolute error loss function L1 is:

[0072]

[0073] Where y is the real all-focus image in the input training set corresponding to the final predicted all-focus image x, N represents the total number of pixels in the image, and x k is the kth pixel in the final predicted full focus image x of the single image depth of field extension model, y k is the pixel x in the true all-focus image y k The corresponding k-th pixel.

[0074] The multi-scale structural similarity loss function and the mean absolute error loss function are mixed to obtain a hybrid loss function and the hybrid loss is calculated for the single image depth of field extension model, so that the final predicted full-focus image x is different from the input real full-focus image.

[0075] Design a mixed loss function to optimize the overall network. The mixed loss function is as follows:

[0076] Loss Mix =λL MS-SSIM +(1-λ)L1

[0077] Where L MS-SSIM is the multi-scale structural similarity loss function, L1 is the mean absolute error loss function, and λ is used to balance the two loss terms and is set to 0.8.

[0078] The hybrid loss function is used to calculate the loss of the single image depth of field extension model and the Adam optimizer is used to adjust the model network structure. The validation set is used to evaluate the performance of the model. The hyperparameters of the model are dynamically adjusted according to the performance on the validation set. The training is repeated until the best hyperparameter combination is found to obtain the final trained single image depth of field extension model.

[0079] S4: Use the final trained single image depth of field extension model to extend the depth of field of the single focus image provided by the test set to obtain an image with extended depth of field.

[0080] In order to further test the feasibility and effectiveness of the method of the present invention, experiments were conducted on the method of the present invention. The present invention uses a no-reference image quality evaluation method and a full-reference image quality evaluation method to evaluate the experimental results of the method of the present invention and other existing single image deblurring and defocus deblurring methods on the proposed single image depth of field extension dataset and the public datasets DPDD and RealDOF.

[0081] The evaluation index of the no-reference image quality evaluation method is time. The evaluation index of the full-reference image quality evaluation method includes pixel mean, standard deviation (SD) and entropy. The pixel mean is used to represent the brightness mean of the image. The larger the value, the brighter the image. The standard deviation is used to measure the discrete degree of the gray value of the image. The larger the value, the higher the contrast of the image. Entropy is used to measure the complexity of the image information. The larger the value, the more details and textures the image contains.

[0082] The evaluation indicators of the full-reference image quality evaluation method include peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and learning-perceptual image patch similarity (LPIPS). PSNR is used to compare the required signal strength with the strength of background noise. The larger its value, the smaller the image noise and the higher the image quality; SSIM reflects the similarity between two images. The higher the SSIM value (ranging from 0 to 1), the more similar the two images are; LPIPS reflects the difference between two images. The lower the LPIPS value, the more similar the two images are.

[0083] We collected 11 sets of unregistered images (including 5 sets of microscopic images) from the Internet, and selected 9 sets of registered images from the Lytro and MFI-WHU datasets. Figure 6 and Figure 7 The comparison results of uncalibrated images and calibrated images under the multi-focus fusion algorithm and the method of the present invention are shown respectively.

[0084] Table 1 shows the evaluation index values ​​of Pixel Mean, SD and Entropy of the all-focus image after depth extension obtained from the experimental results of the depth extension datasets TEST(A) and TEST(B) generated by the present invention using the method of the present invention and other methods. Table 2 shows the evaluation index values ​​of PSNR, SSIM and LPIPS of the all-focus image after depth extension obtained from the experimental results of the public datasets DPDD and RealDOF using the method of the present invention and other methods.

[0085] Table 1 Evaluation index values ​​of depth of field extension results of different methods of the data set provided by the present invention

[0086]

[0087] Table 2 Evaluation index values ​​of depth of field extension results of different methods on DPDD and RealDOF datasets

[0088]

[0089] From the data listed in Table 1, it can be seen that on TEST (A), the present invention is superior to other methods in the evaluation indicators Pixel Mean and Entropy. On TEST (B), Standard Deviation is lower than other algorithms, but the present invention is superior to other methods in other image evaluation indicators Pixel Mean and Entropy. From the data listed in Table 2, it can be seen that on the public data set DPDD, the present invention is superior to other methods in the evaluation indicators PSNR, SSIM and LPIPS. On RealDOF, the SSIM evaluation index value of Uformer is higher than that of the present invention, but the present invention is superior to other methods in other image evaluation indicators PSNR and LPIPS. This shows that the depth of field extension image obtained by the method of the present invention has a higher image quality and is closest to the full-focus image.

[0090] like Figure 4 As shown, the method of the present invention uses the above method to perform a local magnification example of the test results in the data set TEST (B) provided by the method of the present invention, where (a) is a single focus image, (b) is a depth-extended image obtained by the model DRBNet, (c) is a depth-extended image obtained by the model MIMO-UNet, (d) is a depth-extended image obtained by the model MPRNet, (e) is a depth-extended image obtained by the model Uformer, and (f) is a depth-extended image of the method of the present invention. It can be seen from the experimental result diagram that the depth-extended result of the method of the present invention has better quality and contrast than other networks and is closer to the image of the real scene.

[0091] like Figure 5 As shown, the method of the present invention uses the above method to test paired images in the public data sets DPDD and RealDOF, where (a) is a single focus image, (b) is an extended depth of field image obtained for the model MTRNN, (c) is an extended depth of field image obtained for the model MPRNet, (d) is an extended depth of field image obtained for the model MIMO-UNet, (e) is an extended depth of field image obtained for the model DRBNet, (f) is an extended depth of field image obtained for the model Uformer, (g) is an extended depth of field image of the method of the present invention, and (h) is the full-focus image corresponding to (a). The yellow and red areas at the bottom of the figure are the enlarged display effects of the yellow and red areas in the figure, and the values ​​marked in the figure are the PSNR values ​​(unit: dB) calculated from the corresponding algorithm result image and the full-focus image. Figure 4 It can be seen that the method of the present invention enhances the contrast and color of the image, restores the detail information of the image, and makes the enhanced result image closer to the scene in the real world.

[0092] In order to prove that the method of the present invention has the same quality results as multi-image depth of field extension, Figure 6 and Figure 7 The results of the method of the present invention and the multi-focus image fusion method on non-calibrated aligned images and calibrated aligned images are respectively shown. Input is a group of single-focus images, output 1 is the output result after the input images are fused using the multi-focus fusion method, output 2 is the result of processing each image in the input using the method of the present invention, CWT is the fused image obtained by the model CWT, DSIFT is the fused image obtained by the model DSIFT, SEDF is the fused image obtained by the model SEDF, SESF is the fused image obtained by the model SESF, and SDENet is the depth of field extension image of a single input image obtained by the method of the present invention. Figure 5 and Figure 6 It can be seen that the use of non-calibrated aligned images for fusion will result in blurred edges and artifacts, and the image generated by the model CWT also has color deviations. The depth of field extension based on a single image of the method of the present invention can avoid artifacts and color loss caused by fusion, and the experimental aspects of calibrating aligned images can be comparable to the results of the multi-focus fusion algorithm to a certain extent (according to the results of the depth of field extension of a single image, the method of the present invention has better results in images where the main object is more prominent).

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A single image depth of field extension method with deep feature enhancement, characterized in that: Includes steps: S1: Collect and establish a single image depth of field extension dataset, and select a training set, validation set, and test set from the single image depth of field extension dataset; S2: Build a single image depth of field extension model, including an encoder, a Bottleneck structure and a decoder connected in sequence. The encoder, the Bottleneck structure and the decoder all contain a multi-scale feature enhancement structure and an image feature optimization fusion structure connected in sequence. S3: Design a hybrid loss function and use the training set and validation set to train a single image depth of field extension model to obtain a trained depth of field extension model; S4: Use the final trained single image depth of field extension model to extend the depth of field of the single focus image provided by the test set to obtain an image with extended depth of field.

2. The single image depth of field extension method with depth feature enhancement according to claim 1, characterized in that: In step S1, pairs of single-focus images and all-focus images corresponding to the single-focus images in the single-image depth of field extension dataset are used as training sets and verification sets, and single-focus images without all-focus image comparison in the single-image depth of field extension dataset and captured real single-focus images are used as test sets.

3. The single image depth of field extension method with depth feature enhancement according to claim 1 or 2, characterized in that: In the single image depth of field extension model described in step S2, each layer of the encoder and each layer of the decoder respectively includes a residual structure and a multi-scale feature enhancement structure I and an image feature optimization fusion structure I connected in sequence, and the Bottleneck structure includes a multi-scale feature enhancement structure II and an image feature optimization fusion structure II connected in sequence; the image feature optimization fusion structures I and II both include a dynamic sliding multi-head attention module and a deep enhanced feedforward network connected in sequence using a sliding window operation.

4. The single image depth of field extension method with depth feature enhancement according to claim 3, characterized in that: The method for training a single image depth of field extension model described in step S3 is as follows: a training set is input, a single focus image I in the input training set is subjected to a convolution operation to obtain a feature map B1, the feature map B1 is sequentially subjected to a multi-scale feature enhancement structure and an image feature optimization fusion structure of the first layer of the encoder to obtain a high-level feature G1 after the first layer of the encoder enhancement fusion, the high-level feature G1 and the feature map B1 are subjected to a residual connection to obtain deep feature information S1, the deep feature information S1 is downsampled, and sequentially passes through the same encoder layer of M-1 layers to obtain all feature information SM after the single focus image enhancement fusion, the feature information SM is downsampled, and then passes through the multi-scale feature in the Bottleneck structure. The feature enhancement structure and the image feature optimization fusion structure are used to obtain the high-level feature BL after enhancement and fusion in the Bottleneck structure. The high-level feature BL is connected and combined with the feature information SM from the jump connection after upsampling, and then the feature map JM is obtained after the convolution operation. The feature map JM is sequentially passed through the multi-scale feature enhancement structure and the image feature optimization fusion structure, and then the residual link is performed with the feature map JM. Then, the predicted full-focus map YM after decoding by the Mth layer of the decoder is obtained after upsampling. The predicted full-focus map YM passes through the same decoding layer of the M-1 layer in turn, and finally after the convolution operation, the element-by-element addition operation is performed with the single-focus image I in the input training set to obtain the final predicted full-focus map x; A hybrid loss function is used to calculate the loss of the single image depth of field extension model and adjust the model network structure. The validation set is used to evaluate the performance of the model. The hyperparameters of the model are dynamically adjusted according to the performance on the validation set. The training is repeated until the best hyperparameter combination is found to obtain the final trained single image depth of field extension model.

5. The single image depth of field extension method with depth feature enhancement according to claim 4, characterized in that: The calculation process in the multi-scale feature enhancement structures I and II is as follows: the input feature map Fb of each multi-scale feature enhancement structure passes through three pooling layers with different step sizes to extract the global prior information of each layer of the encoder or decoder, and the global prior information extracted by each pooling layer is subjected to a convolution operation, and then further enhanced and optimized by channel attention calculation and spatial attention calculation to obtain enhanced and optimized global prior information. Finally, these enhanced and optimized global prior information are aggregated, passed through a convolution layer, and then combined with the input feature map Fb through a residual to obtain an enhanced global feature F, and the enhanced global feature F is sent to the image feature optimization fusion structure.

6. The single image depth of field extension method with depth feature enhancement according to claim 5, characterized in that: The method for calculating the channel attention is as follows: each input feature map Fa is subjected to global maximum pooling and global average pooling to generate new feature vectors and, then these two feature vectors are passed through a shared multi-layer perceptron MLP to generate two new feature vectors and, then the two feature vectors are added together, and a channel attention map M is generated through a sigmoid activation function. C =sigmoid(M max +M avg ), and finally multiply the channel attention map by the original feature map element by element to obtain the final output feature map F out =M C ⊙Fa; The method for calculating the spatial attention is: output feature map F out After maximum pooling and average pooling, a new feature map F is generated max1 and F avg1 Next, the two feature maps are concatenated to obtain the feature map F concat , and then perform a convolution operation on the concatenated feature map to generate a spatial attention map M S =sigmoid(Conv(F concat )), and finally combine the spatial attention map with the output feature map F out Perform element-by-element multiplication to obtain the final output feature map F out1 =M S ⊙F out .

7. The single image depth of field extension method with depth feature enhancement according to claim 5 or 6, characterized in that: The calculation process in the image feature optimization fusion structure I and II is as follows: the enhanced global feature F is passed through a dynamic sliding multi-head attention module and a deep enhanced feedforward network to obtain enhanced and fused high-level features.

8. The single image depth of field extension method with depth feature enhancement according to claim 7, characterized in that: The calculation process in the dynamic sliding multi-head attention module is: use two deep convolutions to calculate the enhanced global feature F to obtain the query key Sum Where W i (·) is a 1×1 depthwise convolution, It is a 3×3 depth convolution, which divides the enhanced global feature F. For each divided feature window i, the divided query Q is used i , key K i Sum value V i The local attention map of the corresponding size is generated by the softmax activation function; the calculation formula of the local attention map is: where d K is the dimension of the key K, and then the local attention map Attention(Q i ,K i ,V i ) to obtain the attention output map of the complete sequence; the attention output map is residually connected with the enhanced global feature F, and then normalized to obtain the final attention output map; The calculation process in the deep enhanced feedforward network is as follows: the final attention output map passes through a feedforward network containing three layers of deep convolution, the feedforward network output is residually connected with the feedforward network input, and then normalized to obtain enhanced and fused high-level features.

9. The single image depth of field extension method with depth feature enhancement according to claim 8, characterized in that: The method for calculating the loss of a single image depth of field extension model using a hybrid loss function is as follows: using a multi-scale structural similarity loss function to perform structural similarity loss calculation on each layer of the decoder of the single image depth of field extension model, using a mean absolute error loss function to perform mean absolute error loss calculation on the final predicted all-focus image x and the real all-focus image y in the training set, mixing the multi-scale structural similarity loss function and the mean absolute error loss function to obtain a hybrid loss function and performing hybrid loss calculation on the single image depth of field extension model.

10. The single image depth of field extension method with depth feature enhancement according to claim 9, characterized in that: The method for calculating the structural similarity loss is: L MS-SSIM =1-MS-SSIM(x,y) Where M represents the number of decoder layers, MS-SSIM(x,y) represents the multi-scale structural similarity, and l j (x, y) is the brightness similarity between the predicted all-focus image and the corresponding all-focus image at the jth layer of the decoder, c j (x, y) is the contrast similarity between the predicted all-focus image and the corresponding all-focus image at the jth layer of the decoder, s j (x, y) is the structural similarity between the predicted image of the jth layer of the decoder and the corresponding all-focus image; The method for calculating the mean absolute error loss is: Where y is the real all-focus image in the input training set corresponding to the final predicted all-focus image x, N represents the total number of pixels in the image, and x k is the kth pixel in the final predicted full focus image x of the single image depth of field extension model, y k is the pixel x in the true all-focus image y k The corresponding k-th pixel; The method for calculating the mixed loss is: Loss Mix =λL MS-SSIM +(1-λ)L1 Among them, L MS-SSIM is the multi-scale structural similarity loss function, L1 is the mean absolute error loss function, and λ is the parameter used to balance the two loss terms.