Method for Extracting Microscopic Image Blur Kernel and Defocus Restoration Based on Deep Convolutional Network

Abstract

The present invention provides a method for extracting a microscopic image blur kernel and defocus restoration based on a deep convolutional network, which relates to the technical field of image processing. The method first obtains a defocus dataset and preprocesses it; then constructs a blur kernel extraction network model to estimate the blur kernel of the defocus image; constructs a restoration network model, the input of the model is the preprocessed defocus dataset and the estimated blur kernel, and the output is the restored clear image; trains the entire network model composed of the blur kernel extraction network model and the restoration network model, continues to optimize the image reconstruction loss, and ignores the blur kernel reconstruction loss at the same time; uses the trained blur kernel extraction network model and restoration network model to complete the blur kernel estimation and defocus restoration of the microscopic defocus image. This method decomposes the deblurring network into an extraction network for estimating the blur kernel and a restoration network for deblurring the image using the blur kernel, clarifies the influence of the blur kernel during the training of the network, and can improve the accuracy of deblurring.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a method for extracting a microscopic image blur kernel and defocus restoration based on a deep convolutional network. Background Art

[0002] As a precision optical instrument, an optical microscope is widely used in the fields of biomedicine, materials science, industrial manufacturing, etc. However, the depth of field of a high-power optical microscope is very small, and the imaging is prone to defocus blur, which severely limits its use in some high-precision observation fields. Therefore, studying the defocus restoration method for microscopic images is of great significance for improving the imaging resolution of optical microscopes and promoting the wide application of computer vision in the medical field and other fields.

[0003] Defocus restoration refers to the process of inversely solving a clear scene image from a defocus image according to the defocus degradation quantization relationship between the defocus distance and the image blur kernel. Among them, according to whether the blur kernel is known, the defocus restoration method is divided into blind deblurring and non-blind deblurring. The latter effectively shields various noises while improving the imaging resolution, and is currently a relatively mainstream defocus deblurring method for microscopic optical images. However, most of the traditional blur kernel estimation methods based on imaging theory models are complex in model and large in calculation amount, and have low accuracy when dealing with the defocus blur problem of actual microscopic images with aberrations. Compared with the image restoration algorithm based on a complex mathematical model, the method of directly learning the mapping from a defocus image to a clear image using a convolutional neural network skips the process of complex explicit modeling, pairs the defocus image and the clear image and inputs them into the network, and realizes the deblurring process of the input defocus image through network training and optimization. Therefore, it has the advantages of fast speed, high accuracy, and strong robustness. However, the image restoration method based on a convolutional neural network requires that the training data contains both the defocus image of the scene and the corresponding clear image, and this requirement is difficult to meet in practical applications. At the same time, since the blur kernel characteristics that affect the evaluation of the image blur degree are not calculated during the restoration process, in principle, this method is still a blind deblurring method. Therefore, it is difficult to avoid the problems of low accuracy and efficiency and poor robustness existing in conventional blind deblurring methods. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for extracting a microscopic image blur kernel and defocus restoration based on a deep convolutional network to realize the extraction of the blur kernel and defocus restoration of a microscopic image in view of the above-mentioned deficiencies of the prior art.

[0005] To solve the above technical problem, the technical solution adopted by the present invention is: A method for extracting a microscopic image blur kernel and defocus restoration based on a deep convolutional network, comprising the following steps:

[0006] Step 1: Obtain a defocus data set and preprocess it;

[0007] Step 1.1: Obtain a defocus dataset, and uniformly scale each defocus image f(x, y, z) in the defocus dataset to a fixed size. The scaled defocus image is denoted as f H×W×C (x, y, z), where H, W, and C represent the length, width, and number of channels of the image respectively;

[0008] Step 1.2: Label the scaled defocus image f H×W×C (x, y, z) to obtain a defocus dataset; use the image with the smallest defocus distance in each group of defocus images as a label denoted as f(0), and other defocus images are denoted as f(j), where j represents the defocus distance of the image, to obtain a preprocessed defocus dataset F = {f(0), f(j),..., f(J)}, and J is the maximum defocus distance of the image;

[0009] Step 2: Construct a blur kernel extraction network model to estimate the blur kernel of the defocus image; the input dataset of the blur kernel extraction network model is the preprocessed defocus dataset F, and the output is the estimated blur kernel set K;

[0010] Step 2.1: Build a convolutional pooling module; the convolutional pooling module includes six convolutional layers and four pooling layers, and there are two pooling layers after the first and second convolutional layers and after the third and fourth convolutional layers; the input is the defocus image f ∈ R C×H×W , and the output is three different spatial scale feature maps F i , where i = 1, 2, 3;

[0011] Step 2.2: Construct a correlation layer to calculate the cross-correlation of the three different spatial scale feature maps F i to obtain a cross-correlation feature map c i , as shown in the following formula:

[0012] c i = ∑ x，y F i (x - s, y - t)F i (x, y)

[0013] where s and t are both pixel points in different spatial scale feature maps, and the pixel spatial range of s and t is:

[0014] 2 -i m ≤ s, t ≤ 2 -i m

[0015] where m is the spatial dimension of the blur kernel to be restored, and i represents the spatial scale level, taking values of 1, 2, 3; at this time

[0016] Then, for the cross-correlation feature map c iPerform a convolution operation to obtain the convolved feature map C i , as shown in the following formula:

[0017] C i = ReLU(W ci ' * c i + b ci ')

[0018] where W' ci represents the weight of the convolution operation corresponding to c i , and b' ci represents the bias of the convolution operation corresponding to c i . The convolution operations performed on c i all use 1×1 convolution kernels.

[0019] Step 2.3: Construct a blur kernel reconstruction layer. By recursively performing deconvolution operations on the feature maps extracted from relevant layers, obtain the blur kernel;

[0020] First, recursively perform deconvolution operations with a convolution kernel size of 3×3 and a stride of 2 and convolution operations on the feature map C i extracted from the relevant layer at the i-th spatial scale level to obtain the final feature map K. The specific process is as follows:

[0021] k2 = C2 + ReLU(Deconv(C3))

[0022] where Deconv represents a deconvolution operation using 3×3 convolution kernels and a stride of 2, being an intermediate variable for the deconvolution operation;

[0023] K2 = Conv(k2)

[0024] where Conv represents a convolution operation using 1×1 convolution kernels and a stride of 1, being an intermediate variable for the convolution operation;

[0025] k1 = C1 + ReLU(Deconv(K2))

[0026] where, is an intermediate variable for the deconvolution operation;

[0027] K1 = Conv(k1)

[0028] where, is the finally obtained feature map after convolution;

[0029] Integrate the number of channels to be The feature map K1 is gradually reduced to a single-channel feature map K through a series of 3×3 convolutional layers, and this feature map K is the estimated blur kernel;

[0030] Step 2.4: Train the blur kernel extraction network model, use the L1 loss function as the blur kernel reconstruction loss, and minimize the L1 loss function through training;

[0031]

[0032] Among them, K represents the estimated blur kernel, K T represents the true blur kernel, p represents the pixel points of the blur kernel, and P represents the set of all pixel points of the blur kernel;

[0033] Step 3: Construct a restoration network model; the input of the model is the preprocessed defocused dataset F and the estimated blur kernel K, and the output is the restored clear image G;

[0034] Step 3.1: Construct a blur kernel guidance module, map the blur kernel K estimated by the extraction network to a list of weights and biases, and these biases and weights participate in modulating the convolutional output of each layer of the U-net;

[0035] The mapping process of the blur kernel guidance module for the blur kernel K is as follows:

[0036] Z = ReLU(Dense(ReLU(Dense(K))))

[0037] b = Dense(Z)

[0038] w = Dense(Z) + α

[0039] Among them, w and b are the weights and biases mapped by the blur kernel K respectively, α is a trainable bias and is initialized to 1, and Z is an intermediate variable mapped by the blur kernel K;

[0040] Assume that the input of the blur kernel guidance module is r and the output is r′, then the modulation operation of the blur kernel K is as follows:

[0041] rv = ReLU(Conv(r) × w + b)

[0042] Step 3.2: Construct a standard U-net network structure; the input of the standard U-net network structure is the defocused blurred image f, and the output is the restored clear image g;

[0043] The convolutional operation of each layer in the standard U-net network structure is replaced by the blur kernel guidance module constructed in Step 3.1. At this time, the weights and biases mapped by the blur kernel K can participate in modulating the convolutional output of each layer of the U-net network structure;

[0044] Step 3.3: Train the restoration network model; use the L2 loss function as the image reconstruction loss, and minimize the L2 loss function through training;

[0045]

[0046] where g represents the restored clear image, g t represents the in-focus clear image, q represents the pixel points of the image, and Q represents the set of all pixel points in the image;

[0047] Step 4: Train the entire network model composed of the blur kernel extraction network model and the restoration network model;

[0048] In Steps 2 and 3, the extraction network model and the restoration network model are pre-trained respectively to optimize the blur kernel reconstruction loss and the image reconstruction loss; then, the entire network model is trained as a whole to continue optimizing the image reconstruction loss while ignoring the blur kernel reconstruction loss;

[0049] Step 5: Use the trained blur kernel extraction network model and restoration network model to complete the blur kernel estimation and defocus restoration of microscopic defocus images.

[0050] The beneficial effects of adopting the above technical solutions are as follows: The method for extracting the microscopic image blur kernel and defocus restoration based on a deep convolutional network provided by the present invention aims at the problem of unclear images generated during the image acquisition process of a microscope due to factors such as focus failure. The deblurring network is decomposed into an extraction network for estimating the blur kernel and a restoration network for deblurring the image using the blur kernel. During the training of the network, the influence generated by the blur kernel is clarified. This guided non-blind deblurring method can improve the deblurring accuracy. A special correlation layer is introduced into the extraction network to replace the fully convolutional neural network. The Fourier power amplitude component of the blur kernel can be estimated from the autocorrelation analysis of the defocus image, but this non-linearity is difficult to learn through a simple fully convolutional neural network. The method of the present invention introduces this calculation process into the extraction network as a special correlation layer without increasing the number of network layers and the number of filters. In both the extraction network and the restoration network, small convolutional kernels are cascaded to replace large convolutional kernels. Using small convolutional kernels to extract features is suitable for small target samples such as microscopic images. At the same time, to a certain extent, the parameters are reduced, the risk of overfitting is reduced, and it is better trained. A blur kernel guidance module is introduced into the restoration network, mapping the blur kernel to a list of weights and biases to modulate the convolutional output of each layer of the restoration network. After passing through the extraction-restoration network, the defocused image becomes clear, and the tissue structure in the sample also becomes clearly distinguishable. At the same time, the blur kernel can also be accurately restored. In addition, the method has good generalization. When there are changes in the structure or color between samples, the method can still maintain good defocus restoration performance. The images completed with defocus restoration using this method show improvements in both PSNR and SSIM metrics. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 FIG. is a flowchart of the method for extracting the microscopic image blur kernel and defocus restoration based on a deep convolutional network provided by an embodiment of the present invention;

[0052] Figure 2 FIG. is a network architecture diagram of the blur kernel extraction network model provided by an embodiment of the present invention;

[0053] Figure 3 FIG. is a network architecture diagram of the restoration network provided by an embodiment of the present invention;

[0054] Figure 4 FIG. is a network architecture diagram of the blur kernel guidance module provided by an embodiment of the present invention;

[0055] Figure 5 FIG. shows the defocus restoration effect of the test image numbered s16_l1 provided by an embodiment of the present invention. Among them, (a) is the test defocused image, (b) is the defocus restored image, and (c) is the estimated blur kernel;

[0056] Figure 6 Defocus restoration effect of the test image numbered s16_l5 provided by the embodiment of the present invention, where (a) is the defocused test image, (b) is the defocus-restored image, and (c) is the estimated blur kernel;

[0057] Figure 7 Defocus restoration effect of the test image numbered s17_l4 provided by the embodiment of the present invention, where (a) is the defocused test image, (b) is the defocus-restored image, and (c) is the estimated blur kernel;

[0058] Figure 8 Defocus restoration effect of the test image numbered s101_l1 provided by the embodiment of the present invention, where (a) is the defocused test image, (b) is the defocus-restored image, and (c) is the estimated blur kernel;

[0059] Figure 9 Defocus restoration effect of the test image numbered s104_l2 provided by the embodiment of the present invention, where (a) is the defocused test image, (b) is the defocus-restored image, and (c) is the estimated blur kernel. Detailed implementation manners

[0060] The following combines the accompanying drawings and embodiments to further describe in detail the specific implementation manners of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0061] In this embodiment, an experimental environment is built, and the method for microscopic image blur kernel extraction and defocus restoration based on a deep convolutional network of the present invention is used for microscopic image blur kernel extraction and defocus restoration. Among them, the experimental environment: experiments are carried out under the TensorFlow framework. The experimental equipment uses a processor Intel(R) Xeon(R) CPU i7-11700K, with a memory of 64G, an operating system of 64-bit Windows 10, and a GPU model GEFORCE RTX 3080Ti. The experiment runs in GPU mode.

[0062] In this embodiment, the method for microscopic image blur kernel extraction and defocus restoration based on a deep convolutional network, as Figure 1 shown, includes the following steps:

[0063] Step 1: Obtain a defocus dataset and preprocess it;

[0064] Step 1.1: Obtain a defocus dataset, and uniformly scale each defocus image f(x, y, z) in the defocus dataset to a fixed size. The scaled defocus image is represented as f H×W×C (x, y, z), where H, W, and C respectively represent the length, width, and number of channels of the image, in pixels;

[0065] Step 1.2: Mark the defocused image f H×W×C (x, y, z) after scaling to obtain a defocus dataset; take the image with the smallest defocus distance in each group of defocused images as the label denoted as f(0), and other defocused images as f(j), where j represents the defocus distance of the image, to obtain the preprocessed defocus dataset F = {f(0), f(j),..., f(J)}, and J is the maximum defocus distance of the image;

[0066] Step 2: Construct a blur kernel extraction network model as shown in Figure 2 to estimate the blur kernel of the defocused image; the input dataset of the blur kernel extraction network model is the preprocessed defocus dataset F, and the output is the estimated blur kernel set K;

[0067] Step 2.1: Build a convolutional pooling module; the convolutional pooling module includes six convolutional layers and four pooling layers, and there are two pooling layers after the first and second convolutional layers and after the third and fourth convolutional layers; the input is the defocused image f ∈ R C×H×W , and the output is three feature maps F i extracted at different spatial scales, where i = 1, 2, 3;

[0068] First, perform a convolutional operation on the defocused image f, and the convolutional feature map F out1 is expressed as:

[0069] F out1 = ReLU(W0 * f + b0)

[0070] where W0 represents the weight of this convolutional operation, b0 represents the bias of this convolutional operation, this convolutional operation uses n 3×3 convolutional kernels, and F out1 ∈ R n×H×W ;

[0071] Perform another convolutional operation on the feature map F out1 , and the convolutional feature map F1 is expressed as:

[0072] F1 = ReLU(W0' * F out1 + b0')

[0073] where W0' represents the weight of this convolutional operation, b0' represents the bias of this convolutional operation, and this convolutional operation uses 1×1 convolutional kernels,

[0074] Perform a ×2 pooling operation on the feature map F1, and the pooled feature map F'1 is expressed as:

[0075] F'1 = Pooling(F1)

[0076] Among them, Pooling represents a ×2 pooling operation.

[0077] Perform a convolution operation on the feature map F′1, and the convolved feature map F out2 is expressed as:

[0078] F out2 = ReLU(W1* + b1)

[0079] Among them, W1 represents the weight of this convolution operation, b1 represents the bias of this convolution operation, and this convolution operation uses n 3×3 convolution kernels.

[0080] Perform another convolution operation on the feature map F out2 The convolved feature map F2 is expressed as:

[0081] F2 = ReLU(W1′ * F out2 + b1′)

[0082] Among them, W1′ represents the weight of this convolution operation, b1′ represents the bias of this convolution operation, and this convolution operation uses 1×1 convolution kernels.

[0083] Perform a ×2 pooling operation on the feature map F2, and the pooled feature map F′2 is expressed as:

[0084] F′2 = Pooling(F2)

[0085] Among them, Pooling represents a ×2 pooling operation.

[0086] Perform a convolution operation on the feature map F′2, and the convolved feature map F out3 is expressed as:

[0087] F out3 = ReLU(W2* + b2)

[0088] Among them, W2 represents the weight of this convolution operation, b2 represents the bias of this convolution operation, and this convolution operation uses n 3×3 convolution kernels.

[0089] Perform another convolution operation on the feature map F out3 The convolved feature map F3 is expressed as:

[0090] F3 = ReLU(W2′ * F out3 + b2′)

[0091] Among them, W2′ represents the weight of this step of convolution operation, and b2′ represents the bias of this step of convolution operation. This step of convolution operation uses 1×1 convolution kernels,

[0092] Step 2.2: Construct a correlation layer to calculate the cross-correlation of three feature maps F i at different spatial scales to obtain a cross-correlation feature map c i , as shown in the following formula:

[0093] c i =∑ x，y F i (x - s, y - t)F i (x, y)

[0094] where s and t are both pixel points in the feature maps at different spatial scales, and the pixel spatial range of s and t is:

[0095] 2 -i m ≤ s, t ≤ 2 -i m

[0096] where m is the spatial dimension of the blurred kernel to be restored, and i represents the spatial scale level, taking values of 1, 2, 3; at this time

[0097] Then, perform a convolution operation on the cross-correlation feature map c i to obtain a convolved feature map C i , as shown in the following formula:

[0098] C i = ReLU(W ci ′ * c i + b ci ′)

[0099] where W′ ci represents the weight of the convolution operation corresponding to c i , and b′ ci represents the bias of the convolution operation corresponding to c i . The convolution operation performed on c i all uses 1×1 convolution kernels,

[0100] Step 2.3: Construct a blurred kernel reconstruction layer, and obtain the blurred kernel by recursively performing deconvolution operations on the feature maps extracted from the correlation layer;

[0101] First, for the feature map C i extracted from the correlation layer at the i-th spatial scale level,Perform deconvolution and convolution operations recursively with a convolution kernel size of 3×3 and a stride of 2 to obtain the final feature map K. The specific process is as follows:

[0102] k2 = C2 + ReLU(Deconv(C3))

[0103] where Deconv represents a deconvolution operation using convolution kernels of 3×3 and a stride of 2, which is an intermediate variable for the deconvolution operation;

[0104] K2 = Conv(k2)

[0105] where Conv represents a convolution operation using convolution kernels of 1×1 and a stride of 1, which is an intermediate variable for the convolution operation;

[0106] k1 = C1 + ReLU(Deconv(K2))

[0107] where, is an intermediate variable for the deconvolution operation;

[0108] K1 = Conv(k1)

[0109] where, is the finally obtained feature map after convolution;

[0110] Reduce the integrated feature map K1 with the number of channels of to a single-channel feature map K through a series of 3×3 convolutional layers. This feature map K is the estimated blur kernel;

[0111] Step 2.4: Train the blur kernel extraction network model, use the L1 loss function as the blur kernel reconstruction loss, and minimize the L1 loss function through training;

[0112]

[0113] where K represents the estimated blur kernel, K T represents the true blur kernel, p represents the pixel points of the blur kernel, and P represents the set of all pixel points of the blur kernel;

[0114] Step 3: Construct a restoration network model as shown in Figure 3 The input of the model is the preprocessed defocus dataset F and the estimated blur kernel K, and the output is the restored clear image G;

[0115] Step 3.1: Construct a network model as shown in Figure 4The shown blur kernel guidance module maps the blur kernel K estimated by the extraction network into a list of weights and biases, and these biases and weights participate in modulating the convolutional output of each layer of the U-net;

[0116] The mapping process of the blur kernel guidance module for the blur kernel K is as follows:

[0117] Z = ReLU(Dense(ReLU(Dense(K))))

[0118] b = Dense(Z)

[0119] w = Dense(Z)+α

[0120] Among them, w and b are the weights and biases mapped by the blur kernel K respectively, α is a trainable bias and is initialized to 1, and Z is an intermediate variable mapped by the blur kernel K;

[0121] Suppose the input of the blur kernel guidance module is r and the output is r′, then the modulation operation of the blur kernel K is as follows:

[0122] r_ = ReLU(Conv(r)×w + b)

[0123] Step 3.2: Construct a standard U-net network structure; the input of the standard U-net network structure is the defocused blurred image f, and the output is the restored clear image g;

[0124] The convolutional operation of each layer in the standard U-net network structure is replaced by the blur kernel guidance module constructed in Step 3.1. At this time, the weights and biases mapped by the blur kernel K can participate in modulating the convolutional output of each layer of the U-net network structure;

[0125] Step 3.3: Train the restoration network model; use the L2 loss function as the image reconstruction loss, and minimize the L2 loss function through training;

[0126]

[0127] Among them, g represents the restored clear image, g t represents the in-focus clear image, q represents the pixel point of this image, and Q represents the set of all pixel points in this image;

[0128] Step 4: Train the entire network model composed of the blur kernel extraction network model and the restoration network model;

[0129] In Steps 2 and 3, the extraction network model and the restoration network model were pre-trained respectively, optimizing the blur kernel reconstruction loss and the image reconstruction loss; then, the entire network model was trained as a whole to continue optimizing the image reconstruction loss while ignoring the blur kernel reconstruction loss;

[0130] Step 5: Use the trained blur kernel extraction network model and restoration network model to complete the blur kernel estimation and defocus restoration of microscopic defocus images.

[0131] In this embodiment, the network model is trained using the TensorFlow framework. The training is carried out using the ADAM optimizer, with an initial learning rate of 10^4. When the decline of the loss function stagnates for 5 epochs, the learning rate is reduced by 20%. In the blur kernel extraction network, the number of channels of the convolutional module in each spatial scale level of the convolutional pooling module is set to 64 and 32 respectively, the number of channels of the convolutional module in the relevant layer is set to 32, and the number of channels of the convolutional module in the blur kernel reconstruction layer is set to 32. In the restoration network, the number of channels of each layer of the encoder and decoder is set to 128.

[0132] This embodiment is carried out on a publicly available defocus cell image dataset. The dataset contains approximately 130,000 defocus images with different defocus distances, consisting of a training set and a test set. Among them, the training set contains nearly 130,000 defocus images, and the defocus distance corresponding to each defocus image is given. During the training process, approximately 5,000 images are divided from the training set as the validation set, and according to the feedback of the model's performance on the validation set, the structure and parameters of the model are further adjusted. The model structure and parameters with the best performance on the validation set are selected as the final result.

[0133] In this embodiment, the defocus restoration effect and the estimated blur kernel of some test images with different numbers are as Figures 5 - 9 shown. After passing through the extraction-restoration network, the defocused images become clear, and the tissue structures in the samples also become clearly distinguishable. At the same time, the blur kernel can be accurately restored. In addition, it can be seen that the generalization of this method is good. When there are changes in the structure or color between samples, this method can still maintain good defocus restoration performance. Furthermore, by comparing the defocus-restored images with the in-focus clear images, there is not much difference in image quality, which does not affect the observation of cells and subsequent diagnosis, verifying the effectiveness of this method. Table 1 quantifies the indicators of the defocus restoration effects of different algorithms on the test images. The comparison results show that the images defocus-restored using this method perform better in terms of the PSNR and SSIM indicators.

[0134] Table 1 Comparison of defocus restoration results of the same test images under different models

[0135]

[0136]

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for extracting the blur kernel and defocus restoration of microscopic images based on a deep convolutional network, characterized in that: It includes the following steps: Step 1: Obtain a defocus dataset and preprocess it; Step 2: Construct a blur kernel extraction network model to estimate the blur kernel of the defocus image; the input dataset of the blur kernel extraction network model is the preprocessed defocus dataset F, and the output is the estimated blur kernel set K; Step 2.1: Build a convolutional pooling module; the convolutional pooling module includes six convolutional layers and four pooling layers, with two pooling layers after the first and second convolutional layers and two pooling layers after the third and fourth convolutional layers; the input is the defocused image f ∈ R C×H×W , where H, W, and C represent the length, width, and number of channels of the image respectively, and the output is three feature maps F i extracted at different spatial scales, where i = 1, 2, 3; Step 2.2: Construct relevant layers and calculate the cross-correlation of three feature maps F at different spatial scales to obtain the cross-correlation feature map c i ; then perform a convolution operation on the cross-correlation feature map c i to obtain the convolved feature map C i ; i ; Step 2.3: Construct a blur kernel reconstruction layer, and obtain the blur kernel by recursively performing deconvolution operations on the feature maps extracted from relevant layers; Step 2.4: Train the blur kernel extraction network model, use the L1 loss function as the blur kernel reconstruction loss, and minimize the L1 loss function through training; Step 3: Construct a restoration network model; the input of the model is the preprocessed defocus dataset F and the estimated blur kernel K, and the output is the restored clear image G; Step 3.1: Construct a blur kernel guidance module to map the blur kernel K estimated by the extraction network into a list of weights and biases, and these biases and weights participate in modulating the convolution output of each layer of the U-net; Step 3.2: Construct a standard U-net network structure; The input of the standard U-net network structure is the defocus blurred image f, and the output is the restored clear image g; Step 3.3: Train the restoration network model; use the L2 loss function as the image reconstruction loss, and minimize the L2 loss function through training; Step 4: Train the entire network model composed of the blur kernel extraction network model and the restoration network model, continue to optimize the image reconstruction loss, and ignore the blur kernel reconstruction loss at the same time; Step 5: Use the trained blur kernel extraction network model and restoration network model to complete the blur kernel estimation and defocus restoration of the microscopic defocus image.

2. The method for extracting a microscopic image blur kernel and defocus restoration based on a deep convolutional network according to claim 1, wherein: The specific method of Step 1 is as follows: Step 1.1: Obtain a defocus dataset and uniformly scale each defocus image f(x, y, z) in the defocus dataset to a fixed size. The defocus image after scaling is denoted as f H×W×C (x, y, z), where H, W, and C represent the length, width, and number of channels of the image, respectively; Step 1.2: Mark the defocused image f H×W×C (x, y, z) after scaling to obtain a defocus dataset; use the image with the smallest defocus distance in each group of defocused images as the label denoted as f(0), and other defocused images as f(j), where j represents the defocus distance of the image, to obtain the preprocessed defocus dataset F = {f(0), f(j), …, f(J)}, and J is the maximum defocus distance of the image.

3. The method for extracting a microscopic image blur kernel and defocus restoration based on a deep convolutional network according to claim 2, characterized in that: The cross-correlation of the three different spatial scale feature maps F i mentioned in Step 2.2 is calculated to obtain the cross-correlation feature map c i , as shown in the following formula: c i = ∑ x,y F i (x - s, y - t)F i (x, y) Among them, s and t are both pixel points in feature maps of different spatial scales, and the pixel spatial ranges of s and t are: 2 -i m ≤ s, t ≤ 2 -i m where m is the spatial dimension of the blurred kernel to be restored, and i represents the spatial scale level, taking values of 1, 2, and 3; at this time Perform another convolution operation on the cross-correlation feature map c i to obtain the convolved feature map C i , as shown in the following formula: C i = ReLU(W ci ' * c i + b ci ') Among them, W′ ci represents the weight of the convolution operation corresponding to c i , and b′ ci represents the bias of the convolution operation corresponding to c i . The convolution operations performed on c i all use 1×1 convolution kernels, 4. The method for extracting a microscopic image blur kernel and defocus restoration based on a deep convolutional network according to claim 3, characterized in that: The specific method of Step 2.3 is as follows: First, for the feature map C extracted from the relevant layer at the i-th spatial scale level i perform deconvolution and convolution operations recursively with a convolution kernel size of 3×3 and a stride of 2 to obtain the final feature map K; the specific process is as follows: k2 = C2 + ReLU(Deconv(C3)) Among them, Deconv represents the deconvolution operation using 3×3 convolutional kernels with a stride of 2, is an intermediate variable for the deconvolution operation; K2 = Conv(k2) Among them, Conv represents a convolution operation using 1×1 convolution kernels with a stride of 1, which is an intermediate variable for the convolution operation; k1 = C1 + ReLU(Deconv(K2)) Among them, is an intermediate variable for the deconvolution operation; K1 = Conv(k1) Among them, is the final feature map obtained after convolution; The integrated feature map K1 with the number of channels being is gradually reduced to a single-channel feature map K through a series of 3×3 convolutional layers, and this feature map K is the estimated blur kernel.

5. The method for extracting a microscopic image blur kernel and defocus restoration based on a deep convolutional network according to claim 4, wherein: The L1 loss function described in Step 2.4 is shown in the following formula Among them, K represents the estimated blur kernel, and K T represents the true blur kernel, p represents the pixel of the blur kernel, and P represents the set of all pixels of the blur kernel.

6. The method for microscopic image blur kernel extraction and defocus restoration based on a deep convolutional network according to claim 1, characterized in that: The mapping process of the blur kernel guidance module for the blur kernel K is as follows: Z = ReLU(Dense(ReLU(Dense(K)))) b = Dense(Z) w = Dense(Z) + α Among them, w and b are the weights and biases mapped by the blur kernel K respectively, α is a trainable bias and is initialized to 1, and Z is an intermediate variable mapped by the blur kernel K; Set the input of the blur kernel guidance module to r and the output to r′, then the modulation operation of the blur kernel K is as follows: r′ = ReLU(Conv(r) × w + b) The convolution operation of each layer in the standard U-net network structure is replaced by the blur kernel guidance module constructed in Step 3.

1. At this time, the weights and biases mapped from the blur kernel K can participate in modulating the convolution output of each layer of the U-net network structure; The L2 loss function is shown in the following formula: Among them, g represents the restored clear image, and g t represents the clear image at the quasi-focus. q represents the pixel point of this image, and Q represents the set of all pixel points in this image.

Images