A picture deblurring method based on gated wavelet

Abstract

The application discloses a picture deblurring method based on a gating wavelet, which adopts a basic framework of a generative adversarial network, is designed with a gating wavelet processing module, and respectively processes a space domain and a frequency domain of a picture, so that a clear and readable picture with clear edges is obtained, and a better deblurring effect is obtained. Compared with a deblurring model using a blur kernel estimation method, the application has a large receptive field due to an end-to-end hierarchical structure, can remove uneven motion blurs, and solves the problem that the blur kernel estimation method can only model an overall motion direction. The GAN network is used to more easily save texture details in an image, create an image closer to reality, and make people have a higher acceptance for the generated picture in a sense. Since the network is designed based on a CNN, the model has a small size, is easy to deploy to an embedded terminal, and has a faster inference speed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, security monitoring and the like, and specifically relates to a picture deblurring method based on a gated wavelet, which is mainly used for processing the motion blurring problem in pictures and filtering out blurred edges caused by fast-moving objects from the spatial domain and the frequency domain to obtain clear pictures. BACKGROUND

[0002] A harsh natural environment can affect the pictures saved by a recording system, making them blurred. If the blurred pictures are captured in a traffic monitoring network, the license plate may not be recognized, which reduces the law enforcement efficiency. For photographers or astronomers, it is very regrettable if the pictures are blurred due to accidents when they have been waiting for hours to capture a moment. In the fields of photography, videography, astronomical observation and the like, picture information is often lost due to various internal (diffraction, lens chromatic aberration, anti-aliasing filter and the like) or external (object movement, camera shaking, defocus, atmospheric turbulence and the like) factors, resulting in blurring. Although the problem can be alleviated by upgrading the hardware, it is too expensive to replace a batch of high-performance cameras for large-scale road network monitoring equipment. In order to overcome this problem, a more economical and convenient way is to use a deblurring algorithm to recover the pictures. In practical applications, the deblurring algorithm is often deployed on an embedded terminal, such as an automatic access system of a parking lot. Under the condition of limited hardware computing resources, a small algorithm needs to be designed to ensure the accuracy, and therefore a CNN-based network becomes the first choice. SUMMARY

[0003] In order to solve the problem that the final image is blurred due to object movement or shaking of an imaging device during imaging of an optical device, the application provides a picture deblurring method based on a gated wavelet. The method adopts a basic structure of a generative adversarial network, designs a gated wavelet processing module, and processes the spatial domain and the frequency domain of the picture respectively, so that a good deblurring effect is obtained.

[0004] The technical solution of the application to solve the technical problem is to design a picture deblurring method based on a gated wavelet, characterized in that the method comprises the following steps:

[0005] Step 1: Obtain an image training set

[0006] Divide the large-size blurred pictures and the corresponding clear pictures into a plurality of small-size picture blocks of the same size respectively to obtain a blurred picture block set and a clear picture block set; and perform data augmentation on the blurred picture block set and the clear picture block set respectively to expand the size of the data set, so as to obtain a blurred image training set and a clear image training set;

[0007] Step 2: Establish an image deblurring neural network

[0008] The basic architecture of an image deblurring neural network is a generative adversarial network, consisting of a generator and a discriminator. The generator's input is a blurred image, and its processing of the blurred image is as follows: Let the input blurred image be represented as I... R First, I R Intermediate features are extracted from the Inception pre-trained model input into the generator to obtain four feature maps of different scales: F1, F2, F3, and F4. These four feature maps are then fed into four gated wavelet convolutional modules to perform frequency domain learning on the feature maps respectively.

[0009] In the gated wavelet convolution module, the input feature map is first subjected to a two-dimensional discrete wavelet transform, which decomposes it into four sub-bands LL, HL, LH, and HH. Then, projection convolution is applied to each of the four sub-bands, with a kernel size of 1, a stride of 1, padding of 0, and the number of channels being twice that of the input sub-band. The four tensors obtained after projection convolution are then divided into two tensors of equal size along the feature dimension, denoted as LL1, LL2, HL1, HL2, LH1, LH2, HH1, and HH2, respectively. LL2, HL2, LH2, and HH2 are then compressed to 0-1 using the Sigmoid function, serving as gates for their corresponding sub-bands. Finally, LL1, HL1, and LH1 are... HH1 are respectively input into the fifth feature learning module M5, the sixth feature learning module, the seventh feature learning module, and the eighth feature learning module for feature learning. Then, they are multiplied by the corresponding sub-frequency band gates to selectively filter out and retain frequency domain information, resulting in four sub-frequency band feature maps. Finally, wavelet inverse transform is applied to them to restore the size of the input feature map of the gated wavelet convolution module. The fifth, sixth, seventh, and eighth feature learning modules have the same structure, all of which are obtained by sequentially connecting the third convolutional layer, the third ReLU activation function layer, the fourth convolutional layer, the fourth ReLU activation function layer, and the fifth convolutional layer. The output of the previous layer is the input of the next layer.

[0010] For the outputs of the four gated wavelet convolutional modules, feature learning is performed on the feature maps at different scales using the first, second, third, and fourth feature learning modules, respectively. These four feature learning modules have the same structure, consisting of a first convolutional layer, a first ReLU activation function layer, a second convolutional layer, and a second ReLU activation function layer sequentially, with the output of the previous layer serving as the input to the next. The feature learning modules map high-dimensional and low-dimensional features at different scales to a unified feature dimension. Then, the outputs of the second, third, and fourth feature learning modules are upsampled using the second, third, and fourth upsampling layers, respectively, to ensure that the width and height of these three feature maps match the output of the first feature learning module. These three feature maps are then concatenated along the feature dimension. Finally, the convolutional operation of the fusion layer maps the concatenated feature dimension back to the feature dimension of the output feature map from the first feature learning module. After the fusion layer, the output is upsampled again to restore it to the original input image size of the generator, and a residual connection is performed with the original input image to obtain the final output image, which is the restored image obtained from the blurred image.

[0011] The discriminator employs a fully convolutional architecture. First, the input image is passed through a first convolutional layer and a first Leaky ReLU activation function layer, mapping the input features from 3D to 64D. The first convolutional layer has a kernel size of 4, with a stride and padding of 2, resulting in the input spatial domain image. Then, the input spatial domain image is transformed into a frequency domain image of equal size using a two-dimensional discrete Fourier transform. Next, the eighth and ninth feature learning modules are used to extract features from the input spatial and frequency domain images, respectively. The eighth and ninth feature learning modules have identical structures, both consisting of a sixth convolutional layer, a first normalization layer, a second Leaky ReLU activation function layer, a seventh convolutional layer, a second normalization layer, a third Leaky ReLU activation function layer, an eighth convolutional layer, and a third normalization layer. The first layer, the fourth LeakyReLU activation function layer, and the ninth convolutional layer are sequentially connected. The input data is first fed into the sixth convolutional layer, and the output of the previous layer becomes the input of the next layer. The output of the ninth convolutional layer is the output of the corresponding feature learning module. The kernel size of the sixth and seventh convolutional layers is 4, the stride is 2, and the padding is 2. The kernel size of the eighth and ninth convolutional layers is 4, the stride is 1, and the padding is 2. After feature extraction by the eighth and ninth feature learning modules, the input spatial and frequency domain images are mapped into an N×N feature map. The two feature maps are then concatenated to obtain a combined feature map for region evaluation. Each value in the combined feature map represents the evaluation of a region in the original image. The output of the discriminator contains evaluation metrics in both the spatial and frequency domains.

[0012] Step 3: Train the image deblurring neural network

[0013] Step 3.1 Initialize the image deblurring neural network. The initial weights of the convolutional layers are set to conform to a normal distribution with a mean of 0 and a variance of 0.02; the initial weights of the BN layers are set to conform to a normal distribution with a mean of 1 and a variance of 0.02; the initial bias is set to 0; the number of training iterations is set to 300; and the learning rate is set to 0.0001.

[0014] Step 3.2 Input a batch of blurred images from the blurred image training set into the initialized generator. Each blurred image in the batch is first processed by the Inception pre-training layer within the generator to extract features, outputting four feature maps F1, F2, F3, and F4 of different sizes. Then, each feature map is input into a gated wavelet convolution module to decompose the feature map into four sub-bands. The fifth, sixth, seventh, and eighth feature learning modules are used to learn each sub-band. After that, a gating matrix is ​​used to filter the output, and finally, an inverse wavelet transform is performed. The feature maps are restored. The data processed by the gated wavelet convolution module is fed into the first feature learning module, the second feature learning module, the third feature learning module, and the fourth feature learning module to learn the image in the spatial domain. Then, for feature maps of different sizes, upsampling layers with different ratios are used to adjust the feature maps to the same size, and then they are merged in the feature dimension. The number of channels is fused to the number of channels of the initial input image through a fusion layer. Then, an upsampling layer is used to restore the image size to the initial input size, and it is added to the initial input image to obtain a restored image output by the generator.

[0015] Step 3.3: Input a batch of restored images output by the generator in Step 3.2 into the initialized discriminator. After processing through multiple convolutional layers of the discriminator, the image size is gradually reduced, finally obtaining a combined feature map of region evaluation. The evaluation values ​​of the combined feature maps of region evaluation of the restored images are averaged to obtain the evaluation value of the restored image. Then, the clear images corresponding to the batch of blurred images in the clear image training set are input into the discriminator, and the evaluation values ​​of their combined feature maps of region evaluation are averaged to obtain the evaluation value of the corresponding clear image. Then, gradient penalty is calculated for the clear image and the restored image. The specific process is as follows: First, a random number 'a' with a value ranging from 0 to 1 is generated. Then, the tensor of the restored image is multiplied by 1 - 'a', and the tensor of the clear image is multiplied by 'a'. Finally, the two are fused. A tensor sample x is generated according to a random number ratio. Then, the tensor sample x is input into the discriminator to obtain the region evaluation combined feature map y output by the discriminator. The derivative of the tensor sample x and the region evaluation combined feature map y is calculated, and its L2 norm is subtracted by 1. Finally, the maximum value between this value and 0 is taken as the gradient penalty value. The difference between the evaluation value of the clear image and the evaluation value of the restored image is added to the weighted gradient penalty value as the loss of the discriminator for that restored image. Based on the average loss of the discriminator for a batch of restored images, the discriminator's parameters are updated once using backpropagation through the discriminator's optimizer. Then, the parameters of the discriminator's optimizer are updated once, completing one iteration of parameter updates. The training of the discriminator for a batch of blurred images is then complete.

[0016] Step 3.4 Keeping the discriminator parameters unchanged, input a batch of blurred images from the blurred image training set into the initialized generator to obtain the corresponding restored images output by the generator; input the restored images into the discriminator to obtain the evaluation value of the restored images, and the negative value of the evaluation value is the adversarial loss of the generator; calculate the perceptual loss of the restored image and the corresponding blurred image. The perceptual loss calculation process is to use the VGG network pre-trained model to extract features from the restored image and the clear image respectively, and calculate the mean square error of the two feature maps obtained. This value is the perceptual loss value of the clear image and the restored image; the loss of the generator for a blurred image is the sum of the adversarial loss and the perceptual loss; based on the mean value of the generator loss of a batch of blurred images, perform gradient backpropagation on the generator, update the generator parameters and learning rate once, and the training of the generator for a batch of blurred images is completed. At this point, the discriminator and the generator have completed the training of a batch of images.

[0017] Step 3.5: Use the parameters of the discriminator and generator when they completed the training of the previous batch of images as the initial parameters for the training of the next batch of images. Repeat steps 3.2-3.4 until all batches of images have been trained, and the discriminator and generator complete one iteration of training. Use the parameters of the discriminator and generator when they completed the previous iteration of training as the initial parameters for the next iteration of training. Repeat this process until the number of iterations reaches a preset value. Save the parameters of the discriminator and generator when they completed the last iteration of training, and you will get the trained image deblurring neural network model.

[0018] Step 4: Image Deblurring

[0019] The size of the blurred image to be processed is adjusted to the size of the small patch in step one, and then it is input into the image deblurring neural network model trained in step three. The generator outputs the restored image, and then the restored image is restored back to the size of the blurred image to be processed, thus obtaining the deblurred image.

[0020] Compared with existing technologies, the advantages of this invention include: This invention uses generative adversarial networks (GANs) to restore images blurred by factors such as object motion and imaging device jitter, resulting in images with clear edges and high readability. Compared to deblurring models using blur kernel estimation methods, this invention, due to its end-to-end hierarchical structure, has a larger receptive field, can remove uneven motion blur, and solves the problem that blur kernel estimation methods can only model the overall motion direction. Using GANs makes it easier to preserve texture details in images and create more realistic images, making the generated images more acceptable to humans. Because the network is designed based on CNNs, the model size is small, easy to deploy on embedded terminals, and the inference speed is faster. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the basic architecture and principle of an image deblurring neural network according to an embodiment of a gated wavelet-based image deblurring method of the present invention.

[0022] Figure 2 This is a flowchart illustrating the basic architecture and principle of the image deblurring neural network generator, which is an embodiment of the image deblurring method based on gated wavelets of the present invention.

[0023] Figure 3 This is a flowchart illustrating the basic architecture and principle of the gated wavelet convolution module of the image deblurring neural network generator, which is an embodiment of the image deblurring neural network generator of the present invention.

[0024] Figure 4This is a flowchart illustrating the basic architecture and principle of the discriminator in an image deblurring neural network, which is an embodiment of an image deblurring method based on gated wavelets according to the present invention.

[0025] Figure 5 This is a deblurring effect image obtained using an embodiment of the image deblurring method based on gated wavelets of the present invention. Figure 5 (a) in the image is a blurred image. Figure 5 (b) in the image is a deblurred image. Figure 5 (c) in the image is the clear image corresponding to the blurry image.

[0026] Figure 6 This is a deblurring effect image obtained using another embodiment of the image deblurring method based on gated wavelets of the present invention. Figure 6 (a) in the image is a blurred image. Figure 6 (b) in the image is a deblurred image. Figure 6 (c) in the image is the clear image corresponding to the blurry image. Detailed Implementation

[0027] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0028] This invention provides an image deblurring method based on gated wavelets, the method comprising the following steps:

[0029] Step 1: Obtain the image training set

[0030] A large, blurred image and its corresponding sharp image are each divided into several smaller patches of the same size, resulting in a blurred patch set and a sharp patch set. Data augmentation is then performed on both sets to expand the dataset size, resulting in a blurred image training set and a sharp image training set. Data augmentation methods can include operators such as translation, cropping, and color transformation. Alternatively, image processing software such as ImageNet or CCPD can be used to apply blurring algorithms, randomly generating a PSF path and randomly selecting the motion direction and amplitude to simulate jitter. In one embodiment, the size of the smaller patch is 128×128. The blurred image is obtained by processing the sharp image using the PSF path.

[0031] Step 2: Establish an image deblurring neural network

[0032] The basic architecture of an image deblurring neural network is a Generative Adversarial Network (GAN), which includes a generator and a discriminator. The generator's input is a blurred image, and the generator's processing of the blurred image is as follows: Let the input blurred image be represented as I... RBatch size set to 16, I R ∈R 16×3×128×128 R represents the set of real numbers, i.e., I R Let I be the set of real numbers with a size of 16×3×128×128. R Intermediate features are extracted from the Inception pre-trained model input to the generator, resulting in four feature maps at different scales: F1, F2, F3, and F4. These four feature maps are then fed into four gated wavelet convolutional modules to perform frequency domain learning on each feature map.

[0033] In the gated wavelet convolution module, the input feature map is first subjected to a two-dimensional discrete wavelet transform, resulting in four sub-bands: LL, HL, LH, and HH. Then, projection convolution (P1, P2, P3, and P4 in the attached diagram) is applied to each of these four sub-bands. The kernel size is 1, the stride is 1, padding is 0, and the number of channels is twice that of the input sub-bands. Taking the first layer as an example, an input of size 16×3×128×128 undergoes a two-dimensional discrete wavelet transform to obtain four sub-bands of size 16×3×64×64. After applying projection convolution to each sub-band, the size of each sub-band becomes 16×6×64×64. The four tensors obtained after projection convolution are then divided equally along the feature dimension into two tensors of size 16×3×64×64, denoted as LL1, LL2, HL1, HL2, LH1, LH2, HH1, and HH2, respectively. The values ​​of LL2, HL2, LH2, and HH2 are compressed to 0-1 using the Sigmoid function (denoted as σ in the attached diagram) and used as gates for their corresponding sub-frequency bands. LL1, HL1, LH1, and HH1 are then input into the fifth feature learning module M5, the sixth feature learning module M6, the seventh feature learning module M7, and the eighth feature learning module M8, respectively, for feature learning. Afterward, they are multiplied by the corresponding sub-frequency band gates, selectively filtering and retaining frequency domain information to obtain four sub-frequency band feature maps of size 16×3×64×64. Finally, an inverse wavelet transform is applied to these maps to restore them to the size of the input feature map of the gated wavelet convolution module. The fifth feature learning module M5, the sixth feature learning module M6, the seventh feature learning module M7, and the eighth feature learning module M8 have the same structure. They are all formed by sequentially connecting the third convolutional layer C3, the third ReLU activation function layer R3, the fourth convolutional layer C4, the fourth ReLU activation function layer R4, and the fifth convolutional layer C5. The output of the previous layer is the input of the next layer.

[0034] For the outputs of the four gated wavelet convolutional modules, feature learning is performed on the feature maps output at different scales using the first feature learning module M1, the second feature learning module M2, the third feature learning module M3, and the fourth feature learning module M4, respectively. These four feature learning modules have the same structure, each consisting of a first convolutional layer C1, a first ReLU activation function layer R1, a second convolutional layer C2, and a second ReLU activation function layer R2, sequentially connected. The output of the previous layer becomes the input of the next layer. Through these feature learning modules, high-dimensional and low-dimensional features at different scales are mapped to a unified feature dimension. Then, the outputs of the second feature learning module M2, the third feature learning module M3, and the fourth feature learning module M4 are upsampled using the second upsampling layer U2, the third upsampling layer U3, and the fourth upsampling layer U4, respectively, so that the width and height dimensions of these three feature maps are consistent with the output of the first feature learning module M1. These three feature maps (labeled O2, O3, and O4 in the attached diagram) are then concatenated along the feature dimensions. Finally, the concatenated feature dimensions are mapped to the feature dimensions of the output feature map of the first feature learning module M1 (labeled O1 in the attached diagram) through a convolution operation in the fusion layer. After the fusion layer, the output is upsampled again to restore it to the size of the generator's original input image, and a residual concatenation is performed with the original input image to obtain the final output image, which is the restored image obtained from the blurred image.

[0035] The discriminator employs a fully convolutional architecture. First, the input image is passed through a first convolutional layer (P5) and a first Leaky ReLU activation function layer (LR1) to map the input features from 3D to 64D. The kernel size of convolutional layer P5 is 4, with a stride and padding of 2, resulting in the input spatial domain image. Then, the input spatial domain image is transformed into a frequency domain image of equal size using a two-dimensional discrete Fourier transform. Finally, feature extraction is performed on the input spatial domain image and the frequency domain image using the eighth feature learning module (M8) and the ninth feature learning module (M9), respectively. The eighth feature learning module M8 and the ninth feature learning module M9 have the same structure, both consisting of the sixth convolutional layer C6, the first normalization layer N1, the second LeakyReLU activation function layer LR2, the seventh convolutional layer C7, the second normalization layer N2, the third LeakyReLU activation function layer LR3, the eighth convolutional layer C8, the third normalization layer N3, the fourth LeakyReLU activation function layer LR4, and the ninth convolutional layer C9, in sequence. Input data is first fed into the sixth convolutional layer C6, and the output of the previous layer becomes the input of the next layer. The output of the ninth convolutional layer C9 is the output of the corresponding feature learning module. The kernel size of the sixth convolutional layer C6 and the seventh convolutional layer C7 is 4, the stride is 2, and the padding is 2; the kernel size of the eighth convolutional layer C8 and the ninth convolutional layer C9 is 4, the stride is 1, and the padding is 2. The input spatial and frequency domain maps are processed by the eighth feature learning module M8 and the ninth feature learning module M9 to extract features, and are then mapped into an N×N feature map. The two feature maps (labeled O5 and O6 in the attached figure) are then concatenated to obtain a combined feature map for region evaluation. Each value in the combined feature map represents the evaluation of a region in the original map. The output of the discriminator contains evaluation indicators in both the spatial and frequency domains.

[0036] Step 3: Train the image deblurring neural network

[0037] Step 3.1 Initialize the image deblurring neural network. The initial weights of the convolutional layers are set to conform to a normal distribution with a mean of 0 and a variance of 0.02. The initial weights of the BN layers are set to conform to a normal distribution with a mean of 1 and a variance of 0.02. The initial bias is set to 0. The number of iterations is set to 300. The learning rate is set to 0.0001.

[0038] Step 3.2 Input a batch of blurred images from the blurred image training set into the initialized generator. Each blurred image in this batch undergoes feature extraction within the generator via an Inception pre-training layer, outputting four feature maps F1 to F4 of different sizes. Each feature map is then input into a gated wavelet convolution module, decomposing it into four sub-bands. Each sub-band is then learned using the fifth feature learning module M5, the sixth feature learning module M6, the seventh feature learning module M7, and the eighth feature learning module M8. The output is then filtered using a gating matrix, and finally, an inverse wavelet transform is used to reconstruct the feature maps. The data processed by the gated wavelet convolution module is then fed into the first feature learning module M1, the second feature learning module M2, the third feature learning module M3, and the fourth feature learning module M4 for spatial learning. For feature maps of different sizes, upsampling layers with different ratios are used to adjust the feature maps to the same size, and then the features are merged along the feature dimension. The number of channels is fused to the number of channels of the initial input image through a fusion layer, and then the image size is restored to the initial input size through an upsampling layer. The restored image is then added to the initial input image to obtain a restored image output by the generator.

[0039] Step 3.3: Input a batch of restored images output by the generator in Step 3.2 into the initialized discriminator. After processing through multiple convolutional layers of the discriminator, the image size is gradually reduced, finally obtaining a combined feature map of region evaluation. The evaluation values ​​of the combined feature maps of the region evaluation of the restored images are averaged to obtain the evaluation value of the restored image. Then, the clear images corresponding to the batch of blurred images in the clear image training set are input into the discriminator, and the evaluation values ​​of their combined feature maps of region evaluation are averaged to obtain the evaluation value of the corresponding clear image. Then, gradient penalty is calculated for the clear image and the restored image. The specific process is as follows: First, a random number 'a' with a value ranging from 0 to 1 is generated. Then, the tensor of the restored image is multiplied by (1-a), and the tensor of the clear image is multiplied by 'a'. Then, the two are fused to obtain a tensor sample 'x' generated according to the random number ratio. Then, the tensor sample 'x' is input into the discriminator to obtain the region evaluation combined feature map 'y' output by the discriminator. The gradient penalty is calculated by differentiating the tensor sample x and the combined feature map y of the region evaluation, calculating its L2 norm and subtracting 1, and finally taking the maximum value between this value and 0. The difference between the evaluation value of the clear image and the evaluation value of the restored image is added to the weighted gradient penalty value as the loss of the discriminator for that restored image. Based on the mean of the discriminator loss of a batch of restored images, the discriminator parameters are updated once using backpropagation using the discriminator optimizer. Then, the parameters of the discriminator optimizer are updated once, completing one iteration of parameter updates. The training of the discriminator for a batch of blurred images is then complete.

[0040] Step 3.4: Keeping the discriminator parameters unchanged, input a batch of blurred images from the blurred image training set into the initialized generator to obtain the corresponding restored images output by the generator. Input this restored image into the discriminator to obtain the evaluation value of the restored image; the negative value of this evaluation value is the adversarial loss of the generator. Calculate the perceptual loss between the restored image and the corresponding blurred image. The perceptual loss calculation process involves using a VGG network pre-trained model to extract features from both the restored image and the clear image, and calculating the mean square error of the two feature maps. This value is the perceptual loss value for the clear image and the restored image. The generator loss for a blurred image is the sum of the adversarial loss and the perceptual loss. Based on the mean value of the generator loss for a batch of blurred images, perform gradient backpropagation on the generator, updating the generator parameters and learning rate once. The training of the generator for a batch of blurred images is complete. At this point, the discriminator and generator have completed the training for a batch of images.

[0041] Step 3.5: Use the parameters of the discriminator and generator when they completed the training of the previous batch of images as the initial parameters for the training of the next batch of images. Repeat steps 3.2 to 3.4 until all batches of images have been trained and the discriminator and generator have completed one iteration of training. Use the parameters of the discriminator and generator when they completed the previous iteration of training as the initial parameters for the next iteration of training. Repeat this process until the number of iterations reaches a preset value. Save the parameters of the discriminator and generator when they completed the last iteration of training. This gives you the trained image deblurring neural network model.

[0042] Loss function of the discriminator D The WGAN-GP loss function is used to alleviate the instability problem in generative adversarial networks during training, and its formula is expressed as follows:

[0043]

[0044] Where D represents the discriminator, G represents the generator, and P represents the... G P represents the set of reconstructed images output by the generator. data D represents the set of corresponding sharp images; D(x2) represents the discriminator's resolution on P. G The evaluation value of a restored image x2. Indicates the discriminator on P G The mean of the evaluation values ​​of all restored images; D(x1) represents the discriminator's evaluation of P. data The rating of a clear image x1 in the image. Indicates the discriminator on P data The mean of the evaluation values ​​for all clear images in the dataset;

[0045] The gradient penalty term has a penalty distribution P.penalty Between P G With P data The gradient penalty is calculated by first generating a random number *a* between 0 and 1. Then, the restored image is multiplied by (1-a), and the clear image is multiplied by *a*, thus generating a sample *x3* according to the random number ratio. This sample is then input into the discriminator to obtain the feature map *y* output by the discriminator. The derivatives of *x3* and *y* are calculated, their L2 norms are subtracted by 1, and finally, the maximum value between this value and 0 is taken, which is the value of the gradient penalty term. λ is a hyperparameter, representing the weight of the gradient penalty in the discriminator's loss.

[0046] The generator's loss is the sum of the adversarial loss and the perceptual loss:

[0047]

[0048] In the formula, P G Let P represent the set of reconstructed images output by the generator, and D(x) represent the discriminator's response to P. G The evaluation value of a restored image x. Indicates the discriminator on P G The negative of the mean of the evaluation values ​​of all restored images;

[0049] The perceptual loss represents the process of generating an image by assuming it is derived from the content map. By iteratively calculating the content loss, the generated image is made to increasingly approximate the content map. In deblurring tasks, the generated image refers to the restored image, while the content map is the sharpened image. This is represented as follows:

[0050]

[0051] Where j represents the number of layers in the selected VGG network, C i H i W j This indicates the size of the feature map output by the last layer of the VGG network. φ represents the feature map extracted by the VGG network from the restored image. j (y) represents the feature map extracted by the VGG network from the sharp image. The perceptual loss is calculated by using the VGG network pre-trained model to extract features from both the restored and sharp images, and then calculating the mean square error of the two feature maps. This value is the perceptual loss value for the sharp and restored images.

[0052] Step 4: Image Deblurring

[0053] The size of the blurred image to be processed is adjusted to the size of the small patch in step one, and then it is input into the image deblurring neural network model trained in step three. The generator outputs the restored image, and then the restored image is restored back to the size of the blurred image to be processed, thus obtaining the deblurred image.

[0054] Figure 5 and Figure 6 These are deblurring effect images of two embodiments of the image deblurring method based on gated wavelets of this invention. In the two sets of images, (a) is the blurred image, (b) is the deblurred image, and (c) is the clear image (GT) corresponding to the blurred image. As can be seen from the images, the clarity of the deblurred image is significantly better than that of the blurred image, the image quality is better, and the peak signal-to-noise ratio (dB) of the deblurred image is significantly improved, demonstrating a significant deblurring effect.

[0055] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A method for image deblurring based on gated wavelets, characterized in that, The method comprises the following steps: Step one: obtaining an image training set The large-size blurred pictures and the corresponding clear pictures are respectively divided into a plurality of small-size blocks of the same size to obtain a blurred block set and a clear block set; the blurred block set and the clear block set are respectively subjected to data augmentation to expand the size of the data set, and a blurred image training set and a clear image training set are obtained; Step two: establishing an image deblurring neural network The basic framework of the image deblurring neural network is a generative adversarial network, including a generator and a discriminator; the input of the generator is a blurred image, and the processing process of the generator on the blurred image is: let the input blurred image be represented as I R , first input I R into the Inception pre-training model of the generator to extract intermediate features, to obtain four feature maps F1, F2, F3 and F4 of different scales; then the four feature maps are respectively sent into four gated wavelet convolution modules to respectively perform frequency domain learning on the feature maps; In the gating wavelet convolution module, first, the input feature map is subjected to two-dimensional discrete wavelet transform, four sub-bands LL, HL, LH and HH are obtained through decomposition, and then projection convolution is used on the four sub-bands, the convolution kernel size is 1, the step is 1, the padding is 0, and the channel number is twice the input sub-band; then the four tensors obtained after the projection convolution are respectively averaged along the feature dimension into two tensors of the same size, which are respectively denoted as LL1, LL2, HL1, HL2, LH1, LH2, HH1 and HH2; the LL2, HL2, LH2 and HH2 among them are compressed to 0-1 using the Sigmoid function, as the gates of the corresponding sub-bands; then LL1, HL1, LH1 and HH1 are respectively input into the fifth feature learning module M5, the sixth feature learning module, the seventh feature learning module and the eighth feature learning module for feature learning, and then multiplied by the corresponding sub-band gates to selectively filter and retain the frequency domain information, to obtain four sub-band feature maps, and finally the wavelet inverse transform is used to restore the size of the input feature map of the gating wavelet convolution module; the fifth feature learning module, the sixth feature learning module, the seventh feature learning module and the eighth feature learning module have the same structure, and are sequentially connected by a third convolution layer, a third ReLU activation function layer, a fourth convolution layer, a fourth ReLU activation function layer and a fifth convolution layer; the output of the previous layer is the input of the next layer. For the outputs of the four gated wavelet convolution modules, the first feature learning module, the second feature learning module, the third feature learning module and the fourth feature learning module are used respectively to learn the features of the feature maps of different scales, and the four feature learning modules have the same structure and are sequentially connected by a first convolutional layer, a first ReLU activation function layer, a second convolutional layer and a second ReLU activation function layer; the high-dimensional and low-dimensional features of different scales are mapped to a unified feature dimension through the feature learning module; then the outputs of the second feature learning module, the third feature learning module and the fourth feature learning module are respectively up-sampled by using the second up-sampling layer, the third up-sampling layer and the fourth up-sampling layer, so that the width and height dimensions of the three feature maps are consistent with the output of the first feature learning module, and then the three feature maps are spliced along the feature dimension, and finally the feature dimension of the spliced feature is mapped to the feature dimension of the output feature map of the first feature learning module through the convolution operation of the fusion layer; after the fusion layer, the output is up-sampled again to restore the original input image size of the generator, and is connected with the original input image in residual connection to obtain the final output image, that is, the restored image obtained from the blurred image; The discriminator adopts a full convolution form, first passes the input image through a first convolutional layer and a first LeakyReLU activation function layer to map the input features from 3 dimensions to 64 dimensions, wherein the convolution kernel size of the first convolutional layer is 4, the step and the padding are both 2, to obtain the input spatial domain image; then the input spatial domain image is converted to a frequency domain image of the same size through two-dimensional discrete Fourier transform; then the eighth feature learning module and the ninth feature learning module are used to extract features from the input spatial domain image and the frequency domain image respectively; the eighth feature learning module and the ninth feature learning module have the same structure and are sequentially connected by a sixth convolutional layer, a first normalization layer, a second LeakyReLU activation function layer, a seventh convolutional layer, a second normalization layer, a third LeakyReLU activation function layer, an eighth convolutional layer, a third normalization layer, a fourth LeakyReLU activation function layer and a ninth convolutional layer; the input data is first input to the sixth convolutional layer, the output of the previous layer is the input of the next layer, and the output of the ninth convolutional layer is the output of the corresponding feature learning module; the sixth convolutional layer and the seventh convolutional layer have a convolution kernel size of 4, a step of 2 and a padding of 2; the eighth convolutional layer and the ninth convolutional layer have a convolution kernel size of 4, a step of 1 and a padding of 2; after the input spatial domain image and the frequency domain image are extracted by the eighth feature learning module and the ninth feature learning module, they are respectively mapped into an N*N feature map, and then the two feature maps are spliced to obtain a combined feature map for regional evaluation, and each value in the combined feature map represents an evaluation of a region in the original image; the output of the discriminator includes the evaluation indexes of the spatial domain and the frequency domain; Step three: training the image deblurring neural network Step 3.1 initializes the image deblurring neural network, and the initial values of the weights of the convolution layer are set to conform to a normal distribution with a mean of 0 and a variance of 0.02; the initial values of the weights of the BN layer are set to conform to a normal distribution with a mean of 1 and a variance of 0.02; the initial bias is set to 0; the number of iterations of training is set to 300 times; and the learning rate is set to 0.0001; Step 3.2 inputs a batch of blurred images in the blurred image training set to the initialized generator, each blurred image in the batch is first subjected to feature extraction through the Inception pre-training layer in the generator to output four feature maps F1, F2, F3 and F4 of different sizes; then each feature map is input to the gated wavelet convolution module to decompose the feature map into four subbands, and the fifth feature learning module, the sixth feature learning module, the seventh feature learning module and the eighth feature learning module are used to learn each subband, after which the output is filtered using the gating matrix, and finally the feature map is restored through wavelet inverse transformation; the data processed by the gated wavelet convolution module is sent to the first feature learning module, the second feature learning module, the third feature learning module and the fourth feature learning module for learning in the spatial domain, after which the feature maps of different sizes are adjusted to the same size using up-sampling layers of different scales, and then the feature maps are merged in the feature dimension; the number of channels is fused to the number of channels of the initial input image through the fusion layer, and then the image size is restored to the initial input size through an up-sampling layer, and the initial input image is added to obtain a restored image output by the generator; Step 3.

3. Input a batch of restored images output by the generator in step 3.2 to the initialized discriminator, and gradually reduce the image size through the processing of multiple convolutional layers of the discriminator, to finally obtain a regional evaluation combined feature map; take the average of the evaluation values of the regional evaluation combined feature map of the restored image, to obtain the evaluation value of the restored image; then input the clear images in the clear image training set corresponding to the batch of blurred images to the discriminator, take the average of the evaluation values in the regional evaluation combined feature map, to obtain the evaluation value of the corresponding clear image; then, calculate the gradient penalty for the clear image and the restored image, the specific process being: first generate a random number a with a value range of 0-1, then multiply the tensor of the restored image by 1-a and the tensor of the clear image by a, and then fuse the two to obtain a tensor sample x generated according to the proportion of the random number; then input the tensor sample x to the discriminator to obtain the regional evaluation combined feature map y output by the discriminator; after derivation of the tensor sample x and the regional evaluation combined feature map y, calculate the two-norm and subtract 1, and finally take the maximum value between the value and 0, to obtain the value of the gradient penalty; the difference between the evaluation value of the clear image and the evaluation value of the restored image is added to the value of the weighted gradient penalty, as the loss of the discriminator for the restored image; according to the average of the loss of the discriminator for a batch of restored images, perform backward gradient propagation using the optimizer of the discriminator to update the parameters of the discriminator once, then perform a parameter update on the optimizer of the discriminator, complete an iterative update of the parameters of the discriminator, and complete the training of the discriminator for a batch of blurred images; Step 3.

4. Keep the parameters of the discriminator unchanged, input a batch of blurred images in the blurred image training set to the initialized generator to obtain the corresponding restored image output by the generator; input the restored image to the discriminator to obtain the evaluation value of the restored image, and the negative of the evaluation value is the adversarial loss of the generator; calculate the perceptual loss of the restored image and the corresponding blurred image, the calculation process of the perceptual loss being to use the VGG network pre-trained model to extract features from the restored image and the clear image respectively, and calculate the mean square error of the two obtained feature maps, which is the perceptual loss value of the clear image and the restored image; the loss of the generator for one blurred image is the sum of the adversarial loss and the perceptual loss; according to the average of the loss of the generator for a batch of blurred images, perform backward gradient propagation on the generator to update the generator parameters and the learning rate once, complete the training of the generator for a batch of blurred images, and thus the discriminator and the generator complete the training of a batch of images. Step 3.5, the parameters of the discriminator and the generator when the training of the previous batch of images is completed are taken as the initial parameters for the training of the next batch of images, and the process of steps 3.2-3.4 is repeatedly performed until all batches of images are trained, and one iteration of the training of the discriminator and the generator is completed; the parameters of the discriminator and the generator when the previous iteration of the training is completed are taken as the initial parameters for the next iteration of the training, and the iteration is repeatedly performed until the number of iterations reaches a preset value, and the parameters of the discriminator and the generator when the last iteration of the training is completed are saved, and the trained image deblurring neural network model is obtained; Step four: image deblurring processing The size of the blurred image to be processed is adjusted to the size of the small-size image block in step one, and then the blurred image is input into the trained image deblurring neural network model in step three, the recovered image is output by the generator, and the recovered image is restored to the size of the blurred image to be processed, and the deblurred image is obtained.

2. The method of claim 1, wherein the method is based on a gated wavelet. In step one, the blurred image is obtained by processing the clear image through the PSF path.

3. The method of claim 1, wherein the method is based on a gated wavelet. In step one, the size of the small-size image block is 128x128.

4. The method of claim 1, wherein the method is based on a gated wavelet. The loss of the generator is the sum of the adversarial loss and the perceptual loss: where P G denotes the set of reconstructed images generated by the generator, D(x) denotes the evaluation value of a reconstructed image x by the discriminator, G denotes the evaluation value of all reconstructed images by the discriminator, G denotes the negative of the average of the evaluation values of all reconstructed images by the discriminator;​ represents the perceptual loss, which is represented as follows: where j denotes the number of layers of the selected VGG network, C j H j W j denotes the size of the feature map output by the last layer of the VGG network, denotes the feature map extracted by the VGG network on the restored image, φ j (y) denotes the feature map extracted by the VGG network on the clean image.

5. The method of claim 1, wherein the method is based on a gated wavelet. For the input blurred image I of the generator R , the batch size is set to 16, I R ∈R 16×3×128×128 , R represents the real number set, that is, I R is a real number set with a size of 16x3x128x128.

Images