Image restoration device and method based on double estimation network

Abstract

The application discloses an image restoration device and method based on a double estimation network, mainly used for turbulent blur image restoration and improving image definition. The method comprises the following steps: establishing a training data set; improving a multiple-input multiple-output U-net network; realizing multi-scale prediction of a turbulent blur kernel; and improving network robustness by using a Wiener fitting loss. The application realizes that the double estimation multiple-input multiple-output U-net network is used for turbulent blur image restoration, and the method improves the peak signal-to-noise ratio and definition of the restored image.

Description

Technical Field

[0001] This invention relates to an image restoration device and method based on a dual estimation network, characterized by the network simultaneously estimating the sharp image and the blur kernel, and is applied to the restoration of aircraft images degraded by turbulence, belonging to the field of image restoration in image processing. Background Technology

[0002] Atmospheric turbulence severely degrades image quality and reduces its visual value. Image restoration provides a technique for extracting potentially sharp images from turbulent-degraded images. However, this is a classic problem. Given a degraded image, there are infinitely many possible pairings between the potential image and the blur kernel.

[0003] To obtain a unique sharp solution, traditional methods use image priors to constrain the problem. Inspired by the success of convolutional neural networks (CNNs) in image processing tasks, CNNs have been used for image restoration. Early explorations typically applied CNNs to learn blur kernels, then used non-blind image restoration algorithms to estimate the sharp image. These methods reflect the traditional maximum a posteriori (MAP) image restoration framework. Subsequently, researchers attempted to train networks to infer the sharp image directly from the degraded image.

[0004] Figure 2 This is a schematic diagram of the Multiple Input Multiple Output (MIMO) U-Net model, which processes blurred images and generates deblurred images at different scales. It effectively fuses the multi-scale information flow between the encoder and decoder, thereby improving deblurring performance. This model is very effective for single-image motion blur removal tasks. However, currently there is no MIMO neural network model that simultaneously infers the potentially sharp image and the blur kernel. This paper aims to address this issue. Figure 2 Based on the model shown, an improved and optimized dual-estimation image restoration network for atmospheric turbulence degradation was designed. Summary of the Invention

[0005] The technical problem this invention aims to solve is: addressing the issue that existing image restoration methods cannot fully utilize blur kernel information and thus cannot accurately estimate the blur kernel. This invention proposes a dual estimation network that simultaneously estimates both the sharp image and the blur kernel. Using the same training set and training strategy, this method can effectively improve the network's ability to estimate potentially sharp images.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0007] An image restoration device based on a dual estimation network, wherein the backbone network of the device is a U-net-like network structure, and the backbone network contains four types of network modules: encoder module, asymmetric feature fusion module, decoder module, and dual estimation module;

[0008] The encoder module consists of multiple convolutional layers and residual modules, which are used to extract the difference features between blurred and clear images.

[0009] The asymmetric feature fusion module is used to fuse the features extracted by the encoder module at different scales as part of the input to the non-low-level scale decoder module.

[0010] The decoder module is used to obtain the difference features between blurred and sharp images learned by the encoder module;

[0011] The dual estimation module takes the decoder output and the blurred image at the current size as input, and outputs an estimate of the sharp image at the current size.

[0012] This invention also proposes an image restoration method based on a dual estimation network, the method comprising the following steps:

[0013] Step 1: Downsample each image in the publicly available dataset FGVC-aircraft. The downsampling ratio is an integer, defined as the minimum length or width of each image that is greater than 256 but less than 512. [] represents rounding down, m and n represent the length and width of the image, respectively. The generated downsampled images are classified into training and test sets. The training set contains nine-tenths of the images, and the test set contains the remaining one-tenth. Using the von Kalman statistical screen model, 500 blur kernel images with a size of 256×256 are generated with different D / r0 ratios, where D represents the aperture diameter and r0 represents the atmospheric coherence diameter. A total of N turbulent degradation blur kernels are generated, generated by the von Kalman statistical screen model, and their specific definitions are as follows:

[0014] h(x,y)=|F -1 {A(u,v)e jω(u,v)}| 2

[0015] Where h(x, y) is the atmospheric turbulence blur kernel function, A(u, v) is the pupil function of the imaging system, w(u, v) is the random phase masking function, and F -1 This represents the inverse Fourier transform, where j is the imaginary unit;

[0016] Similarly, the generated blur kernel images are classified into training and test sets. The training set contains nine-tenths of the blur kernel images, and the test set contains the remaining one-tenth. During training, the input image of the training set is randomly selected and a random block is selected to obtain a 256×256 image block. This block is then convolved with a blur kernel randomly selected from the blur kernel training set to generate a blur image. The blur image is then fed into the first layer encoder module based on the dual estimation network, and the corresponding clear image and blur kernel serve as the labels for the dual estimation module.

[0017] Step 2: Construct a template-update-based dual estimation network, which includes four convolutional network structures: an encoder module, an asymmetric feature fusion module, a decoder module, and a dual estimation module;

[0018] The encoder module consists of multiple convolutional layers and residual modules, which are used to extract the difference features between blurred and clear images.

[0019] The asymmetric feature fusion module is used to fuse the features extracted by the encoder module at different scales as part of the input to the non-low-level scale decoder module.

[0020] The decoder module is used to obtain the difference features between blurred and sharp images learned by the encoder module;

[0021] The dual estimation module takes the decoder output and the blurred image at the current size as input, and outputs an estimate of the sharp image at the current size.

[0022] Step 3: Train the dual estimation network based on the training dataset; set the weights of the empirical Fourier loss and Wiener loss to 0.1 and 0.01 respectively, and set δ to 0.01. The loss function is:

[0023]

[0024] Where S represents the number of downsampling layers in the U-net network, The network estimates the image at layer s, I s Represents a clear image of the target in layer s. The network estimates the fuzzy kernel at layer s, k s N represents the target blur kernel of layer s, s F represents the total number of pixels in the current s-layer target image, F(·) represents the Fourier transform, and ||·||1 represents the L1 norm. -1 Represents the inverse Fourier transform, B s The input blurred image represents layer s. Represents F(k) s The conjugate of ) and δ is the signal-to-noise ratio.

[0025] The training process uses the Adam optimizer with a learning rate of 0.00001. It trains for a total of 200 rounds, with 90,000 generated blurred images trained in each round. The batch size is set to 32, and the learning rate is halved every 50 rounds.

[0026] Step 4: Complete the restoration process of the actual blurred image; load the trained weight parameters and configuration file, configure the dual estimation network; send the blurred image into the first layer encoder module network, estimate and output the predicted clear image and blur kernel.

[0027] Compared with existing technologies, the beneficial effects of this invention are as follows: During output, the output module is replaced with a dual estimation module to predict the blur kernel that causes image blurring, thereby enhancing the network's feature extraction and prediction capabilities. The addition of Wiener fitting loss reduces the impact of noise on the network's prediction ability and improves the network's ability to learn intrinsic image information. This invention estimates the blur kernel while restoring the image, helping users further analyze image blur information and assess the network's restoration capability, thus improving the accuracy and robustness of the restoration method. This invention overcomes the problem that existing methods cannot simultaneously predict potentially sharp images and blur kernels. Under the same training data and training strategy, the network using this invention... Figure 2 The baseline network improved the PSNR evaluation metric from 32.49 to 32.69 on the test dataset. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the implementation of the dual estimation network for turbulence image reconstruction in this invention.

[0029] Figure 2 A basic principle block diagram of the multi-input multi-output U-net restoration model;

[0030] Figure 3 This is a block diagram illustrating the basic principle of the dual estimation network of this invention;

[0031] Figure 4 This is a schematic diagram of a portion of the network structure blocks in the dual estimation network of the present invention;

[0032] Figure 5 This is a comparison of the restoration effect of the present invention on simulated turbulence images;

[0033] Figure 6 This is a comparison of the restoration effect of the present invention on real turbulence images. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0035] This invention discloses an image restoration method based on a dual estimation network. The method utilizes a dual estimation module to simultaneously predict the potentially sharp image and the blur kernel.

[0036] To train the aforementioned dual estimation network, a novel training loss, the Wiener fitting loss, was invented. This loss is defined as follows:

[0037]

[0038] Based on experience, the weighting parameters λ of Fourier loss and Wiener loss are... F , λ W and λ k Set them to 0.1, 0.01, and 1 respectively, and set δ to 0.01.

[0039] The proposed restoration method based on a dual estimation network effectively utilizes fuzzy kernel information. The newly proposed Wiener fitting loss can effectively reduce the impact of noise.

[0040] The network framework of this invention is as follows: Figure 3 As shown, it mainly consists of four convolutional network structures: an encoder module, an asymmetric feature fusion module, a decoder module, and a dual estimation module. The network results of these four modules are as follows:

[0041] The encoder module is divided into a first-layer encoder module, a second-layer encoder module, and a third-layer encoder module.

[0042] The first-layer encoder module consists of a convolutional layer with C input channels (C=1 for grayscale images, C=3 for RGB images), 32 output channels, and a 3×3 kernel, connected in series with eight residual modules, each with 32 input and 32 output channels. The structure of the residual modules is as follows: Figure 4 As shown in (a), the structure of the residual module mainly includes two convolutional layers with a stride of 1 and a kernel size of 3×3. The output of this module is equal to the input plus the result of the input passing through the two convolutional layers.

[0043] The second-layer encoder module first performs feature encoding on the output of the first-layer encoder and the downsampled image, respectively. Then, a feature attention module further refines the features, and finally, the extracted features are further encoded. The network structure for feature encoding the first-layer encoder output is a convolutional layer with 32 input channels, 64 output channels, a stride of 2, and a kernel size of 3×3. A shallow feature extraction module processes the downsampled image, and its convolutional module structure is as follows... Figure 4 As shown in (b), the shallow feature extraction module mainly consists of five convolutional layers, each with a stride of 1. The kernel sizes are 3×3, 1×1, 3×3, 1×1, and 1×1, respectively. The output of this module is equal to the input concatenated with the results of the first four convolutional layers and then passed through the last convolutional layer. The number of output channels is 32, 64, 64, 64-C, and 64, respectively. Then, the output of the shallow feature extraction module is used to apply feature attention to the output of another module. The feature attention module is as follows: Figure 4As shown in (c), the feature attention module structure includes a convolutional layer with a stride of 1 and a kernel size of 3×3. Its output is equal to the pixel-by-pixel sum of the encoder module output and the residual module output multiplied by the encoder module output and the result passed through the convolutional layer. The feature attention module convolutional layer has 64 output channels. Then, it passes through eight residual modules, each with 64 input and output channels, to obtain the output of the second encoder module. The structure of the residual modules is the same as that of the first encoder module.

[0044] The third-layer encoder module has the same structure as the second-layer encoder module. First, feature encoding is performed on both the output of the second-layer encoder and the downsampled image. Then, a feature attention module is used to further refine the features. Finally, the extracted features are further encoded. The network structure for feature encoding the output of the second-layer encoder is a convolutional layer with 64 input channels, 128 output channels, a stride of 2, and a kernel size of 3×3. A shallow feature extraction module is used to process the downsampled image; its convolutional module structure is as follows... Figure 4 As shown in (b), the number of output channels is 64, 128, 128, 128-C, and 128, respectively. Then, the output of the shallow feature extraction module is used to perform feature attention on the output of another module. The feature attention module is shown below. Figure 4 As shown in (c), the output channels of the convolutional layer of the feature attention module are 128. Then, the output of the third encoder module is obtained through eight residual modules, each with 128 input and output channels. The structure of the residual modules is the same as that in the first encoder module.

[0045] The input to the asymmetric feature fusion module is the output of the three-layer encoder module, and its feature structure diagram is as follows: Figure 4 As shown in (d), the asymmetric feature fusion module consists of three scaling operations plus two convolutional layers with a stride of 1 and a kernel size of 3×3. The output of this module is the result of feature concatenation of the output of the three-layer encoder module after scaling, followed by two convolutional layers. Internally, this module can be further divided into a second-layer asymmetric fusion module and a first-layer asymmetric feature fusion module, whose outputs serve as part of the second-layer decoder and the first-layer decoder, respectively, with 64 and 32 output channels.

[0046] The decoder module is divided into a first-layer decoder module, a second-layer decoder module, and a third-layer decoder module.

[0047] The first-layer decoder module consists of a convolutional layer with 64 input channels, 32 output channels, and a 3×3 kernel, as well as eight residual modules with 32 input and 32 output channels each. Its input is the concatenation of the output of the first-layer non-feature fusion module and the output of the second-layer decoder module.

[0048] The second-layer decoder module consists of a convolutional layer with 128 input channels, 64 output channels, and a 3×3 kernel; eight residual modules with 64 input and 64 output channels each; and a deconvolutional layer with 64 input channels and 32 output channels, a 4×4 kernel, and a stride of 2. Its input is the concatenation of the outputs from the second-layer non-feature fusion module and the third-layer decoder module.

[0049] The third-layer decoder module consists of eight residual modules, each with 128 input and 128 output channels, and one deconvolutional layer with 128 input and 64 output channels, a 4×4 kernel size, and a stride of 2. Its input is the output of the third-layer encoder module.

[0050] The dual estimation module includes a first-level dual estimation module, a second-level dual estimation module, and a third-level dual estimation module.

[0051] The first dual estimation module consists of two independent convolutional layers with input and output channels of 32 and C respectively, and a kernel size of 3×3. The output of one convolutional layer is added pixel-by-pixel to the blurred image to obtain a potential sharp image estimate. The output of the other convolutional layer is the blur kernel estimate.

[0052] The second-layer dual estimation module consists of two independent convolutional layers with input and output channels of 64 and C respectively, and a kernel size of 3×3. The output of one convolutional layer is added pixel-by-pixel to the blurred image downsampled once to obtain a potential sharp image estimate at half the size. The output of the other convolutional layer is the blur kernel estimate at half the size.

[0053] The third dual estimation module consists of two independent convolutional layers with input and output channels of 128 and C respectively, and a kernel size of 3×3. The output of one convolutional layer is added pixel-by-pixel to the blurred image downsampled twice to obtain a potential sharp image estimate at 1 / 4 size. The output of the other convolutional layer is the blur kernel estimate at 1 / 4 size.

[0054] Unless otherwise specified, the stride of the convolutional layers mentioned above is set to 1 by default.

[0055] Based on the aforementioned restoration method using a dual estimation network, the image restoration inference process performed by the network during prediction is as follows: Figure 1 As shown. Figure 1The input blurred image is sequentially processed through the following main modules to obtain a sharp image estimate: a first-layer encoder module, a second-layer encoder module, a third-layer encoder module, a third-layer decoder module, a second-layer decoder module, a first-layer decoder module, and a first-layer dual estimation module. Half-sampled and quarter-sampled images of the input blurred image are used as inputs to the second-layer and third-layer encoders, respectively. The outputs of all encoder modules are used in the asymmetric feature fusion module, and the outputs of the asymmetric feature fusion module are used as input features to the second-layer and third-layer decoder modules, respectively.

[0056] Build with PyTorch Figure 3 The network structure framework shown below, using training and testing on a simulated dataset as an example, will be further described in detail below with reference to the accompanying drawings, illustrating the specific implementation of the image restoration method based on a dual estimation network of the present invention. Figure 1 This is a flowchart of the restoration algorithm based on a dual estimation network.

[0057] Step 1: Downsample each image in the publicly available dataset FGVC-aircraft. The downsampling ratio is an integer, defined as the ratio that makes the minimum length or width of each image greater than 256 but less than 512. [] represents rounding down, and m and n represent the length and width of the image, respectively. Downsampling reduces noise and improves the image signal-to-noise ratio. The generated downsampled images are classified into training and test sets. The training set contains nine-tenths of the images, and the test set contains the remaining one-tenth. Using a von Kalman statistical screen model with different D / r0 ratios, 500 256×256 blur kernel images are generated. Similarly, the generated blur kernel images are classified into training and test sets. The training set contains nine-tenths of the blur kernel images, and the test set contains the remaining one-tenth. During training, input images from the training set are randomly selected, and random blocks are selected to obtain 256×256 image blocks. These blocks are then convolved with blur kernels randomly selected from the blur kernel training set to generate blurred images. This ensures batch training of the network. The blurred images are fed as input into the first layer encoder module based on the dual estimation network. The corresponding sharp images and blur kernels serve as labels for the dual estimation module.

[0058] Step 2: Construct a template-update-based convolutional neural network. The dual estimation network comprises four convolutional network structures: an encoder module, an asymmetric feature fusion module, a decoder module, and a dual estimation module. These four modules are described in detail below.

[0059] Step 2.1: Construct the encoder module. The encoder module is divided into a first-layer encoder module, a second-layer encoder module, and a third-layer encoder module.

[0060] Step 2.1.1: The first-layer encoder module consists of a convolutional layer with C input channels (C=1 for grayscale images, C=3 for RGB images), 32 output channels, and a 3×3 kernel size, connected in series with eight residual modules, each with 32 input and 32 output channels. The structure of the residual modules is as follows: Figure 4 As shown in (a).

[0061] Step 2.1.2: The second-layer encoder module first performs feature encoding processing on the output of the first-layer encoder and the downsampled image respectively. Then, a feature attention module is used to further refine the features, and finally, the extracted features are further encoded. The network structure for feature encoding of the first-layer encoder output is a convolutional layer with 32 input channels, 64 output channels, a stride of 2, and a kernel size of 3×3. A shallow feature extraction module is used to process the downsampled image, and its convolutional module structure is as follows... Figure 4 As shown. The number of output channels are 32, 64, 64, 64-C, and 64, respectively. Then, the output of the shallow feature extraction module is used to apply feature attention to the output of another module. The feature attention module is shown below. Figure 4 As shown in (b), the output channels of the convolutional layer of the feature attention module are 64. Then, the output of the second encoder module is obtained through eight residual modules, each with 64 input and output channels. The structure of the residual modules is the same as that in the first encoder module.

[0062] Step 2.1.3: The third-layer encoder module has the same structure as the second-layer encoder module. First, feature encoding is performed on the output of the second-layer encoder and the downsampled image. Then, a feature attention module is used to further refine the features. Finally, the extracted features are further encoded. The network structure for feature encoding the output of the second-layer encoder is a convolutional layer with 64 input channels, 128 output channels, a stride of 2, and a kernel size of 3×3. A shallow feature extraction module is used to process the downsampled image; its convolutional module structure is as follows... Figure 4 As shown. The number of output channels are 64, 128, 128, 128-C, and 128, respectively. Then, the output of the shallow feature extraction module is used to apply feature attention to the output of another module. The feature attention module is shown below. Figure 4 As shown in (c), the output channels of the convolutional layer of the feature attention module are 128. Then, the output of the third encoder module is obtained through eight residual modules, each with 128 input and output channels. The structure of the residual modules is the same as that in the first encoder module.

[0063] Step 2.2: Construct the asymmetric feature fusion module. The input to the asymmetric feature fusion module is the output of the three-layer encoder module, and its feature structure diagram is shown below. Figure 4As shown in (d), this module can be further divided into a second-layer asymmetric fusion module and a first-layer asymmetric feature fusion module. The outputs of these modules serve as part of the second-layer decoder and the first-layer decoder, respectively, with 64 and 32 output channels.

[0064] Step 2.3: Construct the decoder module. The decoder module is divided into a first-layer decoder module, a second-layer decoder module, and a third-layer decoder module.

[0065] Step 2.3.1: The first-layer decoder module consists of a convolutional layer with 64 input channels, 32 output channels, and a 3×3 kernel, and eight residual modules with 32 input and 32 output channels each. Its input is the concatenation of the output of the first-layer asymmetric feature fusion module and the output of the second-layer decoder module.

[0066] Step 2.3.2: The second-layer decoder module consists of a convolutional layer with 128 input channels, 64 output channels, and a 3×3 kernel; eight residual modules with 64 input and 64 output channels each; and a deconvolutional layer with 64 input channels and 32 output channels, a 4×4 kernel, and a stride of 2. Its input is the concatenation of the output of the second-layer asymmetric feature fusion module and the output of the third-layer decoder module.

[0067] Step 2.3.3: The third-layer decoder module consists of eight residual modules, each with 128 input and output channels, and one deconvolution layer with 128 input and 64 output channels, a 4×4 kernel size, and a stride of 2. Its input is the output of the third-layer encoder module.

[0068] Step 2.4: Construct the dual estimation module. The dual estimation module includes a first-level dual estimation module, a second-level dual estimation module, and a third-level dual estimation module.

[0069] Step 2.4.1: The first-layer dual estimation module consists of two independent convolutional layers with input and output channels of 32 and C respectively, and a kernel size of 3×3. The output of one convolutional layer is added pixel-by-pixel to the blurred image to obtain a potential sharp image estimate. The output of the other convolutional layer is the blur kernel estimate.

[0070] Step 2.4.1: The second-layer dual estimation module consists of two independent convolutional layers with input and output channels of 64 and C respectively, and a kernel size of 3×3. The output of one convolutional layer is added pixel-by-pixel to the blurred image downsampled once to obtain a potential sharp image estimate at half size. The output of the other convolutional layer is a blur kernel estimate at half size.

[0071] Step 2.4.1: The third-layer dual estimation module consists of two independent convolutional layers with input and output channels of 128 and C respectively, and a kernel size of 3×3. The output of one convolutional layer is added pixel-by-pixel to the blurred image downsampled twice to obtain a potential sharp image estimate at 1 / 4 size. The output of the other convolutional layer is a blur kernel estimate at 1 / 4 size.

[0072] Unless otherwise specified, the stride of the convolutional layers mentioned above is set to 1 by default.

[0073] Step 3: Train the dual estimation network based on the training dataset. Empirically, the weights of the Fourier loss and Wiener loss are set to 0.1 and 0.01 respectively, and δ is set to 0.01. The loss function is:

[0074]

[0075] Where S represents the number of downsampling layers in the U-net network. The network estimates the image at layer s, I s Represents a clear image of the target in layer s. The network estimates the fuzzy kernel at layer s, k s N represents the target blur kernel of layer s, s F represents the total number of pixels in the current s-layer target image, F(·) represents the Fourier transform, and ||·||1 represents the L1 norm. -1 Represents the inverse Fourier transform, B s The input blurred image represents layer s. Represents F(k) s The conjugate of ) and δ is the signal-to-noise ratio.

[0076] During the training process, this invention uses the Adam optimizer with a learning rate of 0.00001. A total of 200 training rounds are conducted, with 90,000 generated blurred images trained in each round. The batch processing value is set to 32, and the learning rate is halved every 50 rounds.

[0077] Step 4: Complete the actual blurry image restoration process. Load the trained weight parameters and configuration file, and configure the dual-estimation-based network. Feed the blurry image into the first-layer encoder module network. The network prediction steps are as follows: Figure 1 As shown, the algorithm estimates and outputs the predicted sharp image and blur kernel.

[0078] Figure 5 The image on the left shows two blurred images of turbulent degradation from simulation, while the image on the right shows two clear images restored by the present invention.

[0079] Figure 6 The image on the left shows two real, blurry images of turbulent degradation, while the image on the right shows two clear images restored by this invention.

[0080] The parts of this invention not described in detail are well-known to those skilled in the art.

[0081] Those skilled in the art should recognize that the above embodiments are merely illustrative of the present invention and are not intended to limit the present invention. Any variations or modifications to the above embodiments that are within the essential spirit of the present invention will fall within the scope of the claims of the present invention.

Claims

1. An image restoration device based on a dual estimation network, characterized in that, The backbone network of the device is a U-net-like network structure, which contains four types of network modules: encoder module, asymmetric feature fusion module, decoder module, and dual estimation module. The encoder module consists of multiple convolutional layers and residual modules, which are used to extract the difference features between blurred and clear images. The asymmetric feature fusion module is used to fuse the features extracted by the encoder module at different scales as part of the input to the non-low-level scale decoder module. The decoder module is used to obtain the difference features between blurred and sharp images learned by the encoder module; The dual estimation module takes the decoder output and the blurred image at the current size as input, and outputs an estimate of the sharp image at the current size. The dual estimation module includes a first-level dual estimation module, a second-level dual estimation module, and a third-level dual estimation module; The first layer dual estimation module consists of two independent input and output channels, 32 and C respectively, and a 3×3 convolutional layer. The output of one convolutional layer is added pixel by pixel to the blurred image to obtain the potential sharp image estimate, and the output of the other convolutional layer is the blur kernel estimate. The second-layer dual estimation module consists of two independent input and output channels, 64 and C respectively, and convolutional layers with a kernel size of 3×3. The output of one convolutional layer is added pixel by pixel to the blurred image downsampled once to obtain the potential sharp image estimate at 1 / 2 size. The output of the other convolutional layer is the blur kernel estimate at 1 / 2 size. The third-layer dual estimation module consists of two independent input and output channels, 128 and C respectively, and a 3×3 convolutional layer. The output of one convolutional layer is added pixel by pixel to the blurred image downsampled twice to obtain the potential sharp image estimate at 1 / 4 size. The output of the other convolutional layer is the blur kernel estimate at 1 / 4 size. The loss functions of the dual estimation network include image pixel loss, image Fourier loss, and blur kernel pixel loss, which are defined as follows: Where S represents the number of downsampling layers in the U-net network, The network estimates the image at layer s. Represents a clear image of the target in layer s. The network estimates the fuzzy kernel at layer s. N represents the target blur kernel of layer s, s This represents the total number of pixels in the current s-layer target image. Represents Fourier transform, Represents the first norm, For fuzzy kernel loss weights, The weights are Fourier loss weights; the network incorporates Wiener fitting loss to improve its robustness. Wiener fitting loss is defined as follows: in, Represents the inverse Fourier transform. The input blurred image represents layer s. represent conjugate, This refers to the signal-to-noise ratio.

2. The apparatus according to claim 1, characterized in that, The encoder module is divided into a first-layer encoder module, a second-layer encoder module, and a third-layer encoder module; The first layer encoder module consists of a convolutional layer with C input channels (C=1 for grayscale images, C=3 for RGB images), 32 output channels, and a 3×3 kernel size, and eight residual modules with 32 input and 32 output channels connected in series. The second-layer encoder module first performs feature encoding on the output of the first-layer encoder and the downsampled image, respectively. Then, a feature attention module further refines the features, and finally, the extracted features are further encoded. The network structure for feature encoding the first-layer encoder output is a convolutional layer with 32 input channels, 64 output channels, a stride of 2, and a kernel size of 3×3. The downsampled image is processed by a shallow feature extraction module, with output channels of 32, 64, 64, 64-C, and 64 respectively. Then, the output of the shallow feature extraction module is used to apply feature attention to the output of another module. The feature attention module's convolutional layer has 64 output channels. Finally, the output of the second-layer encoder module is obtained through eight residual modules with 64 input and 64 output channels each. The structure of the residual modules is the same as that in the first-layer encoder module. The third-layer encoder module has the same structure as the second-layer encoder module. First, feature encoding is performed on the output of the second-layer encoder and the downsampled image. Then, a feature attention module is used to further refine the features. Finally, the extracted features are further encoded. The network structure for feature encoding of the second-layer encoder output is a convolutional layer with 64 input channels, 128 output channels, a stride of 2, and a kernel size of 3×3. A shallow feature extraction module is used to process the downsampled image, with the output channels being 64, 128, 128, 128-C, and 128 respectively. Then, the output of the shallow feature extraction module is used to apply feature attention to the output of another module. The output channel of the feature attention module's convolutional layer is 128. Finally, the output of the third-layer encoder module is obtained through eight residual modules with 128 input and output channels. The structure of the residual modules is the same as that in the first-layer encoder module.

3. The apparatus according to claim 2, characterized in that, The input to the asymmetric feature fusion module is the output of the three-layer encoder module. This module is internally divided into a second-layer asymmetric fusion module and a first-layer asymmetric feature fusion module. Its output is used as part of the second-layer decoder and the first-layer decoder, respectively, with 64 and 32 output channels.

4. The apparatus according to claim 3, characterized in that, The decoder module is divided into a first-layer decoder module, a second-layer decoder module, and a third-layer decoder module; The first-layer decoder module consists of a convolutional layer with 64 input channels, 32 output channels, and a kernel size of 3×3, as well as eight residual modules with 32 input and 32 output channels each. Its input is the concatenation of the output of the first-layer non-feature fusion module and the output of the second-layer decoder module. The second-layer decoder module consists of a convolutional layer with 128 input channels, 64 output channels, and a 3×3 kernel size; eight residual modules with 64 input and 64 output channels each; and a deconvolutional layer with 64 input and 32 output channels, a 4×4 kernel size, and a stride of 2. Its input is the concatenation of the output of the second-layer non-feature fusion module and the output of the third-layer decoder module. The third-layer decoder module consists of eight residual modules with 128 input and output channels each, and a deconvolution layer with 128 input and 64 output channels, a 4×4 kernel size, and a stride of 2. Its input is the output of the third-layer encoder module.

5. The apparatus according to claim 1, characterized in that, The network training image data was constructed, including blurred images, corresponding sharp images, and corresponding turbulent blur kernels. The sharp images were taken from the public dataset FGVC-aircraft, which contains 10,200 aircraft images of 102 different aircraft models, with 100 images for each model. Nine-tenths of these images were used for network training, and the remaining one-tenth were used for network testing. The turbulent blur kernels were generated using the von Kalman statistical screen model, and their specific definitions are as follows: in, It is an atmospheric turbulence fuzzy kernel function. It is the pupil function of the imaging system. It is a random phase masking function. Represents the inverse Fourier transform. It is the imaginary unit; The random phase screen function is generated by inversion from the von Kalman power spectrum model; by adjusting The value of , where Indicates the aperture diameter. Let represent the coherence diameter of the atmosphere. A total of N turbulence degradation fuzzy kernels are generated. Similarly, nine-tenths of these fuzzy kernels are used for network training, and the remaining one-tenth is used for network testing.

6. An image restoration method based on a dual estimation network, employing the apparatus described in claim 1, characterized in that, The method includes the following steps: Step 1: Downsample each image in the publicly available dataset FGVC-aircraft. The downsampling ratio is an integer, defined as the minimum length or width of each image that is greater than 256 but less than 512. [ ] represents rounding down, m and n represent the length and width of the image, respectively. The generated downsampled images are classified into training and test sets. The training set contains nine-tenths of the images, and the test set contains the remaining one-tenth. Different settings are used with the von Kalman statistical screen model. The ratio generates 500 blurred kernel images with a size of 256×256, of which Indicates the aperture diameter. The coherence diameter of the atmosphere is represented by N turbulence degradation fuzzy kernels, generated by the von Kalman statistical phase screen model, and specifically defined as follows: in, It is an atmospheric turbulence fuzzy kernel function. It is the pupil function of the imaging system. It is a random phase masking function. Represents the inverse Fourier transform. It is the imaginary unit; Similarly, the generated blur kernel images are classified into training and test sets. The training set contains nine-tenths of the blur kernel images, and the test set contains the remaining one-tenth. During training, the input image of the training set is randomly selected and a random block is selected to obtain a 256×256 image block. This block is then convolved with a blur kernel randomly selected from the blur kernel training set to generate a blur image. The blur image is then fed into the first layer encoder module based on the dual estimation network, and the corresponding clear image and blur kernel serve as the labels for the dual estimation module. Step 2: Construct a template-update-based dual estimation network, which includes four convolutional network structures: an encoder module, an asymmetric feature fusion module, a decoder module, and a dual estimation module; The encoder module consists of multiple convolutional layers and residual modules, which are used to extract the difference features between blurred and clear images. The asymmetric feature fusion module is used to fuse the features extracted by the encoder module at different scales as part of the input to the non-low-level scale decoder module. The decoder module is used to obtain the difference features between blurred and sharp images learned by the encoder module; The dual estimation module takes the decoder output and the blurred image at the current size as input, and outputs an estimate of the sharp image at the current size. Step 3: Train the dual estimation network based on the training dataset; set the weights of the empirical Fourier loss and Wiener loss to 0.1 and 0.01 respectively. Set to 0.01, the loss function is: Where S represents the number of downsampling layers in the U-net network, The network estimates the image at layer s. Represents a clear image of the target in layer s. The network estimates the fuzzy kernel at layer s. N represents the target blur kernel of layer s, s This represents the total number of pixels in the current s-layer target image. Represents Fourier transform, Represents the first norm, Represents the inverse Fourier transform. The input blurred image represents layer s. represent conjugate, Signal-to-noise ratio; The training process uses the Adam optimizer with a learning rate of 0.00001. It trains for a total of 200 rounds, with 90,000 generated blurred images trained in each round. The batch size is set to 32, and the learning rate is halved every 50 rounds. Step 4: Complete the restoration process of the actual blurred image; load the trained weight parameters and configuration file, configure the dual estimation network; send the blurred image into the first layer encoder module network, estimate and output the predicted clear image and blur kernel.

7. The method according to claim 6, characterized in that, Step 2 includes, Step 2.1: Construct the encoder module; the encoder module is divided into a first-layer encoder module, a second-layer encoder module, and a third-layer encoder module; Step 2.1.1: The first layer encoder module consists of a convolutional layer with C input channels (C=1 for grayscale images, C=3 for RGB images), 32 output channels, and a 3×3 kernel size, and eight residual modules with 32 input and 32 output channels connected in series. Step 2.1.2: The second-layer encoder module first performs feature encoding processing on the output of the first-layer encoder and the downsampled image respectively. Then, a feature attention module is used to further refine the features. Finally, the extracted features are further encoded. The network structure for feature encoding processing of the first-layer encoder output is a convolutional layer with 32 input channels, 64 output channels, a stride of 2, and a kernel size of 3×3. The downsampled image is processed by a shallow feature extraction module, with the number of output channels being 32, 64, 64, 64-C, and 64 respectively. Then, the output of the shallow feature extraction module is used to perform feature attention on the output of another module. The output channel of the feature attention module's convolutional layer is 64. Then, the output of the second-layer encoder module is obtained through 8 residual modules with 64 input and 64 output channels. The structure of the residual modules is the same as that in the first-layer encoder module. Step 2.1.3: The third-layer encoder module has the same structure as the second-layer encoder module. First, feature encoding is performed on the output of the second-layer encoder and the downsampled image. Then, a feature attention module is used to further refine the features. Finally, the extracted features are further encoded. The network structure for feature encoding of the second-layer encoder output is a convolutional layer with 64 input channels, 128 output channels, a stride of 2, and a kernel size of 3×3. A shallow feature extraction module is used to process the downsampled image, with the output channels being 64, 128, 128, 128-C, and 128 respectively. Then, the output of the shallow feature extraction module is used to apply feature attention to the output of another module. The output channel of the feature attention module's convolutional layer is 128. Finally, the output of the third-layer encoder module is obtained through 8 residual modules with 128 input and output channels. The structure of the residual module is the same as that in the first-layer encoder module.

8. The method according to claim 7, characterized in that, Step 2 also includes, Step 2.2: Construct an asymmetric feature fusion module; The input to the asymmetric feature fusion module is the output of the three-layer encoder module. This module is internally divided into a second-layer asymmetric fusion module and a first-layer asymmetric feature fusion module. Its output is used as part of the second-layer decoder and the first-layer decoder, respectively, with 64 and 32 output channels.

9. The method according to claim 8, characterized in that, Step 2 also includes, Step 2.3: Construct the decoder module, which is divided into a first-layer decoder module, a second-layer decoder module, and a third-layer decoder module; Step 2.3.1: The first layer decoder module consists of a convolutional layer with 64 input channels, 32 output channels, and a kernel size of 3×3, and eight residual modules with 32 input and 32 output channels each. Its input is the concatenation of the output of the first layer asymmetric feature fusion module and the output of the second layer decoder module. Step 2.3.2: The second-layer decoder module consists of a convolutional layer with 128 input channels, 64 output channels, and a kernel size of 3×3; eight residual modules with 64 input and 64 output channels each; and a deconvolutional layer with 64 input and 32 output channels, a kernel size of 4×4, and a stride of 2. Its input is the concatenation of the output of the second-layer asymmetric feature fusion module and the output of the third-layer decoder module. Step 2.3.3: The third layer encoder module consists of eight residual modules with 128 input and output channels each, and a deconvolution layer with 128 input and 64 output channels, a kernel size of 4×4, and a stride of 2. Its input is the output of the third layer encoder module.

10. The method according to claim 9, characterized in that, Step 2 also includes, Step 2.4: Construct a dual estimation module, which includes a first-level dual estimation module, a second-level dual estimation module, and a third-level dual estimation module; Step 2.4.1: The first layer dual estimation module consists of two independent input and output channels, 32 and C respectively, and a convolutional layer with a kernel size of 3×3. The output of one convolutional layer is added pixel by pixel to the blurred image to obtain the latent sharp image estimate, and the output of the other convolutional layer is the blur kernel estimate. Step 2.4.1: The second-layer dual estimation module consists of two independent input and output channels, 64 and C respectively, and convolutional layers with a kernel size of 3×3; the output of one convolutional layer is added pixel by pixel to the blurred image downsampled once to obtain the potential sharp image estimate at 1 / 2 size, and the output of the other convolutional layer is the blur kernel estimate at 1 / 2 size; Step 2.4.1: The third layer dual estimation module consists of two independent input and output channels, 128 and C respectively, and a convolutional layer with a kernel size of 3×3. The output of one convolutional layer is added pixel by pixel to the blurred image downsampled twice to obtain the potential sharp image estimate at 1 / 4 size. The output of the other convolutional layer is the blur kernel estimate at 1 / 4 size.

11. The method according to claim 10, characterized in that, The stride of each convolutional layer is set to 1 by default.

Images