Image deblurring method based on generative adversarial network and storage medium

Abstract

The invention discloses a construction method of a generative adversarial network, which comprises the following steps: adopting a basic architecture based on the generative adversarial network GAN, adopting an FPN feature pyramid structure for a generator, and adopting a CNN convolutional neural network structure for a discriminator; adding a self-attention module in each layer of structure of the FPN to obtain an improved generative adversarial network; wherein the self-attention module receives a feature map from the FPN as input, converts the feature map into a query matrix Q, a key value matrix K and a value matrix V in a matrix space, determines weight distribution of the V by calculating similarity between the Q and the K, and obtains output fused with a self-attention feature vector; the deblurred image quality is high, the training efficiency is high, and the training process is stable.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image deblurring method and a storage medium based on a generative adversarial network. Background Art

[0002] Image restoration aims to restore the original appearance of degraded images from low-quality images. It is one of the most concerned issues in the fields of computer vision, medicine, and military. Image restoration includes image deblurring, image denoising, and image super-resolution. Among them, image deblurring not only enhances the reliability and practicality of image applications such as unmanned driving and security monitoring, but also has important significance for improving image quality and image recognition accuracy.

[0003] In real applications, motion blur caused by camera shake and artifacts caused by the movement of objects relative to the camera reduce the efficiency of image information acquisition and the visual perception quality of the image. Blur is usually caused by unknown and spatially varying blur kernels, and may also be affected by noise and other artifacts from unknown sources. The superposition of these effects complicates image analysis and poses challenges to subsequent computer vision analysis.

[0004] At present, the methods for image deblurring are mainly divided into traditional methods and deep learning methods. Traditional methods regard image deblurring as a deconvolution problem when the blur kernel is known, including Wiener filtering, Gaussian filtering, Lucy-Richardson algorithm, etc. These methods have the advantages of small computational complexity, simplicity, and easy implementation, but if the blur kernel is unknown or affected by noise, it is difficult to process.

[0005] With the rapid development of convolutional neural networks (CNNs), deep learning methods have been widely used in image blur processing. Compared with traditional methods, deep learning methods have the advantages of real-time, scalability, and efficiency. For example, Su et al. used CNN to predict motion blur kernels for image deblurring for the first time, but this method was inefficient when processing some nonlinear motion blurred images. Schuler et al. combined the forward mode with CNN to improve the quality and operation speed of deblurred images, but the performance decreased when processing images with more complex blur kernels. Nah et al. restored clear images by constructing multi-scale CNNs, further improving the deblurring effect. Kupyn et al. applied generative adversarial networks (GANs) to image deblurring research, using residual networks as the backbone of adversarial networks, which significantly improved the training speed, but this method only had a good deblurring effect on local images.

[0006] DeblurGAN is an image deblurring algorithm based on generative adversarial networks (GANs). It has the characteristics of dense connections and residual connections, which can effectively reduce information loss in the network and improve the accuracy of the model. DeblurGAN-v2 is an improved version of DeblurGAN. It is built on GAN and can flexibly use different backbone networks to achieve steady improvements in performance and efficiency.

[0007] However, in actual application scenarios, DeblurGAN-v2 still has some problems, such as information loss caused by downsampling, incomplete restoration of image details, etc. Therefore, in order to further improve the quality of deblurred images and network training efficiency, DeblurGAN-v2 can be further improved.

[0008] In addition, the objective function of GAN involves the process of finding the maximum and minimum values, including multiple factors such as the real image, random noise vector, expected value of the real image, and expected value of the generated image. Although GAN has powerful generation capabilities, it often faces problems such as gradient collapse and difficulty in converging to a stable Nash equilibrium during training, which makes GAN training difficult and the quality of the generated images may be unstable. Summary of the invention

[0009] The purpose of the present invention is to provide an image deblurring method and storage medium based on a generative adversarial network to address the problems existing in the prior art, with high quality of deblurred images, high training efficiency and stable training process.

[0010] To achieve the above object, the technical solution adopted by the present invention is:

[0011] A method for building a generative adversarial network comprises the following steps:

[0012] The basic architecture based on the Generative Adversarial Network (GAN) is adopted. The generator adopts the FPN feature pyramid structure, and the discriminator adopts the CNN convolutional neural network structure.

[0013] In each layer of the FPN structure, a self-attention module is added to obtain an improved generative adversarial network;

[0014] Among them, the self-attention module receives the feature map from FPN as input and converts it into the query matrix Q, key-value matrix K and value matrix V in the matrix space. The weight distribution of V is determined by calculating the similarity between Q and K, and the output of the self-attention feature vector is obtained.

[0015] The fusion process of the self-attention module includes:

[0016] The generator inputs a three-channel blurred image, and the shape of the feature map X output by each layer in the bottom-up path of FPN is

[0017]

[0018] Where C is the number of channels of the input image, and N is the set of w*h matrices of the input image;

[0019] By querying the weight matrix w q , key value weight matrix w k , value weight matrix w v , the input feature map X is converted into the query matrix Q, key value matrix K and value matrix V through linear transformation, the expression formula is as follows

[0020] [QKV]=[w q w k w v ]X;

[0021] Use Q and K to determine the weight distribution of V and get the self-attention feature vector output O, which is expressed as follows

[0022]

[0023] Among them, QK T is the dot product of the query matrix Q and the key matrix K, d q is the feature dimension of the query matrix Q, is the scaling factor, and the softmax function is used to convert the similarity score into a probability distribution, i.e., a weight distribution;

[0024] Normalize the output self-attention feature vector O;

[0025] At each layer of FPN, the output feature map is fused with the self-attention feature vector to enhance FPN’s capture of global information and obtain more detailed information.

[0026] Query weight matrix w q , key value weight matrix w k , value weight matrix w v The dimensions are

[0027]

[0028] Among them, C is the input feature dimension, is the output feature dimension.

[0029] Set the loss function of the generator and discriminator;

[0030] The improved generative adversarial network described in any one of claims 1 to 3 is trained using a data set, and optimized multiple times until convergence to obtain a target network.

[0031] The loss function of the discriminator adopts the improved hinge loss function L D , given by the hinge loss function L d And the gradient penalty term GP, its expression formula is

[0032] L D =L d +λGP,

[0033]

[0034] Where D(x) is the output probability of the discriminator for the real image x, D(G(z)) is the output probability of the discriminator for the image G(z) generated by the generator, E represents the expectation, and λ is the weight number; represents the random interpolation between the real sample and the generated sample, Denotes the discriminator's random interpolation The gradient of

[0035] Hinge loss function L d Find the maximum probability and expectation of the probability of the real data D(x) output by the discriminator and the image D(G(z)) generated by the discriminator output, and sum their expectations to optimize the performance of the discriminator;

[0036] The gradient penalty term GP is used to prevent gradient disappearance and gradient explosion problems and make the training process more stable.

[0037] The loss function L of the generator G By the mean square error loss L p , Perceptual loss L adv and content loss L c The composition is expressed as

[0038] L G =λ p *L p +λ c *L c +λ adv *L adv ,

[0039] Among them, λ p , c , adv They represent the mean square error loss L p , Perceptual loss L adv and content loss L c The weight coefficient of .

[0040] The improved generative adversarial network is trained using the dataset and optimized iteratively for multiple times until convergence to obtain the target generative adversarial network, including:

[0041] Select a public dataset for the image deblurring task, including blurred images and corresponding real images;

[0042] Fixed the generator parameters, input the blurred image into the generator, and generated the deblurred image;

[0043] The generated deblurred image and the real image are input into the discriminator together and the discriminator loss function is calculated;

[0044] Use the optimizer to update the parameters of the discriminator to minimize the discriminator loss function;

[0045] Fix the discriminator parameters and input the blurred image into the generator again to generate a new deblurred image.

[0046] Calculate the generator loss function and use the optimizer to update the generator parameters to minimize the generator loss function;

[0047] Repeat the iteration until the loss function converges to a predetermined value or reaches a preset number of iterations to obtain the target network.

[0048] An image deblurring method inputs a blurred image into the target network and outputs a corresponding clear image.

[0049] A computer-readable storage medium stores a computer program, wherein the computer program implements the above method steps when executed by a processor.

[0050] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the above method steps are implemented when the processor executes the computer program.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. By adopting the basic architecture based on the generative adversarial network (GAN), and combining the FPN feature pyramid structure and the CNN convolutional neural network structure, an efficient generator and discriminator are constructed. The FPN structure helps to capture the multi-scale features of the image, while the CNN structure enhances the ability of feature extraction and image generation;

[0053] 2. A self-attention module is added to each layer of the FPN structure to establish a long-distance dependency relationship between the current pixel and other pixels, so as to more effectively capture the local and global features of the image and extract the key details of the image, which helps to generate higher quality deblurred images and reduce information loss and insufficient restoration of image details;

[0054] 3. By introducing the self-attention mechanism, the similarities between different blurred areas in the image can be accurately captured. In the deblurring process, more attention can be paid to the blur itself, rather than simply processing the entire image indiscriminately, thereby making more efficient use of computing resources.

[0055] 4. By improving the discriminator loss function, a gradient penalty term is introduced to prevent the gradient vanishing and gradient exploding problems, making the training process more stable;

[0056] 5. The loss function of the generator consists of mean square error loss, perceptual loss and content loss. By reasonably setting the weight coefficient, the various losses are balanced and the training effect of the generator is optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 Schematic diagram of the conversion process of the self-attention module in the embodiment of the present application;

[0059] Figure 2 Schematic diagram of the fusion process of the self-attention module in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] In the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish descriptions and cannot be understood as indicating or suggesting relative importance. It should also be understood that although the terms "first", "second", etc. are used to describe various elements in some embodiments of the present application in the text, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first table can be named as the second table, and similarly, the second table can be named as the first table without departing from the scope of the various described embodiments. The first table and the second table are both tables, but they are not the same table.

[0062] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0063] The following is an explanation of some terms in this application. It should be noted that these explanations are for the convenience of understanding by those skilled in the art and are not intended to limit the scope of protection claimed in this application.

[0064] Generative Adversarial Network (GAN): Generative Adversarial Network (GAN) is a deep learning model consisting of two neural networks: the generator and the discriminator. The goal of the generator is to generate fake data that is as close to the real data as possible, while the goal of the discriminator is to distinguish between real data and fake data generated by the generator. The two networks are constantly optimized through a process of mutual competition and game until the generator can generate data that is realistic enough that the discriminator has difficulty distinguishing between real and fake data.

[0065] FPN feature pyramid structure: FPN feature pyramid structure is a network structure for extracting multi-scale features. It extracts features from images at different scales and fuses these features to obtain richer and more comprehensive feature information. FPN structure has been widely used in target detection, image segmentation and other fields. In FPN, two paths, top-down and bottom-up, are usually used to fuse features of different scales to obtain more refined feature maps.

[0066] VGG19: VGG19 is a deep convolutional neural network structure consisting of 19 weight layers (including convolutional layers and fully connected layers). It uses very small 3x3 convolution kernels and gradually increases the number of convolutional layers to deepen the network depth.

[0067] DeblurGAN: DeblurGAN is an image deblurring algorithm based on generative adversarial networks (GANs). It uses a neural network structure called Residual-in-Residual Dense Network, which has the characteristics of dense connections and residual connections, and can effectively reduce information loss in the network and improve the accuracy of the model. DeblurGAN-v2 is an improved version of DeblurGAN, built on GAN, and can flexibly use different backbone networks to achieve steady improvements in performance and efficiency.

[0068] In actual application scenarios, DeblurGAN-v2 still has some problems, such as information loss caused by downsampling, incomplete restoration of image details, etc. Therefore, in order to further improve the quality of deblurred images and network training efficiency, DeblurGAN-v2 can be further improved.

[0069] In addition, the objective function of GAN can be expressed as a process of finding the maximum and minimum values, which involves the real image x, the random noise vector z, the expected value D(x) of the real image, and the expected value D(G(z)) of the generated image G(z). Although GAN has powerful generation capabilities, it often faces problems such as gradient collapse and difficulty in converging to a stable Nash equilibrium during training. These problems make GAN training difficult and the quality of the generated images may be unstable.

[0070] In view of the above technical problems, a first aspect of an embodiment of the present application provides a method for building a generative adversarial network, comprising the following steps:

[0071] S1. The basic architecture based on the generative adversarial network GAN is adopted. The generator adopts the FPN feature pyramid structure, and the discriminator adopts the CNN convolutional neural network structure.

[0072] Among them, the basic architecture can adopt the DeblurGAN-v2 model, or other GAN-based models, such as CycleGAN, StyleGAN, etc., which can combine FPN and CNN structures to build generators and discriminators.

[0073] Preferably, the discriminator adopts the VGG19 model, which uses the powerful feature extraction capability and stable network structure of VGG19 to improve the performance and training stability of GAN.

[0074] S2. In each layer of the FPN structure, a self-attention module is added to obtain an improved generative adversarial network.

[0075] By adding a self-attention mechanism to the feature map of each layer and establishing a long-distance dependency relationship between the current pixel and other pixels, it is possible to more effectively capture the local and global features of the image and extract the key details of the image.

[0076] By introducing the self-attention mechanism, we can accurately capture the similarities between different blurred areas in the image. During the deblurring process, we can focus more on the blur itself instead of just processing the entire image indiscriminately, thereby making more efficient use of computing resources.

[0077] Specifically, the self-attention module receives the feature map from FPN as input and converts it into the query matrix Q, key-value matrix K and value matrix V in the matrix space. The weight distribution of V is determined by calculating the similarity between Q and K, thereby obtaining the self-attention feature vector output. The feature map output by the self-attention module is then fused with the feature map of the corresponding layer in FPN to obtain the output of the fused self-attention feature vector. After normalization, this output feature map is sent to the next layer of the feature pyramid network (FPN) for processing.

[0078] In the bottom-up path of FPN, the feature map output by each layer is fused with the self-attention mechanism module. This fusion strengthens FPN's supplement to global information and enables it to obtain more detailed information. Originally, FPN mainly focused on the local detail features of the image when obtaining image details, but after being fused with the self-attention mechanism, FPN can better capture the overall contextual information of the blurred image, improving the ability of the generative adversarial network model to understand the overall blurred image.

[0079] Through the above steps, the improved generative adversarial network can process blurred images more effectively and generate higher quality deblurred images.

[0080] In some embodiments, the fusion process of the self-attention module includes:

[0081] S2.1. Input the three-channel blurred image to the generator. For the feature map X output by each layer in the bottom-up path of FPN, its shape is

[0082]

[0083] Among them, C is the number of channels of the input image, which is 3 in this case, N is the set of input image w*h matrices, which represents the batch size of the input image, h represents the height of the feature map, and w represents the width of the feature map.

[0084] It should be noted that as the FPN bottom-up path goes deeper, the height h and width w of the feature map usually decrease gradually through downsampling operations, while the number of channels C increases through convolution operations. The description here is for the general shape of the feature map X output by each layer, and does not involve specific numerical changes.

[0085] The subsequent conversion of the feature map to the matrix space for calculation is an internal process for implementing the self-attention mechanism, and does not affect the basic shape description of the feature map X in FPN. Therefore, in the bottom-up path of FPN, the feature map X output by each layer still maintains the above shape.

[0086] like Figure 1 As shown in S2.2, by querying the weight matrix w q , key value weight matrix w k , value weight matrix w v , the input feature map X is converted into the query matrix Q, key value matrix K and value matrix V through linear transformation, the expression formula is as follows

[0087] [QKV]=[w q w k w v ]X;

[0088] Use Q and K to determine the weight distribution of V and get the self-attention feature vector output O, which is expressed as follows

[0089]

[0090] Among them, QK T is the dot product of the transpose of the query matrix Q and the key-value matrix K, which is used to calculate the similarity between them, d q is the feature dimension of the query matrix Q, As a scaling factor, to prevent the gradient vanishing problem caused by excessive dot product results, the softmax function is used to convert the similarity score into a probability distribution, i.e., a weight distribution.

[0091] S2.3. Normalize the output self-attention feature vector O.

[0092] like Figure 2 As shown in S2.4, at each layer of FPN, the output feature map is fused with the self-attention feature vector to enhance FPN's capture of global information and obtain more detailed information.

[0093] For example, the feature map output by FPN is added or multiplied element by element with the self-attention feature vector to achieve feature fusion, so that FPN takes into account both local and global information when extracting features.

[0094] The fused feature map will be used as the input of the next layer of FPN or used to generate a deblurred image.

[0095] In the deblurring task, the self-attention mechanism can identify which areas are blurry and which areas are clear by carefully analyzing each area in the image. On this basis, the network will give higher weights to the key feature areas in the blurry areas, which are usually the parts of the blurry image that need to be restored. For other non-critical areas, lower weights will be given to reduce over-processing of these areas.

[0096] Through this weight distribution method, the model can use computing resources more efficiently and focus more computing resources on the key restoration areas of the blurred image. At the same time, since the self-attention mechanism can capture the global information in the image, it can also help the model establish the relationship between local features and global features, so as to better understand the overall context information of the blurred image.

[0097] In some embodiments, the query weight matrix w q , key value weight matrix w k , value weight matrix w v The dimensions are

[0098] Among them, C is the input feature dimension, is the output feature dimension,

[0099] For example, query the weight matrix w q The dimension is Represents mapping the input features from C dimensions to dimensionality, that is, mapping it to a lower dimensional space for processing, and reducing the amount of calculation through dimensionality reduction.

[0100] According to a second aspect of an embodiment of the present application, a training method for a generative adversarial network is provided, comprising the following steps:

[0101] S3. Set the loss functions of the generator and the discriminator to guide the network training process.

[0102] S4. Use the dataset to train the improved generative adversarial network, perform multiple iterations of optimization until convergence, and obtain the target network.

[0103] Through the above training steps, a target generative adversarial network model that can efficiently and accurately remove image blur can be obtained.

[0104] In some embodiments, the loss function of the discriminator adopts the improved hinge loss function L D, given by the hinge loss function L d And the gradient penalty term GP, its expression formula is

[0105] L D =L d +λGP,

[0106]

[0107]

[0108] Where D(x) is the output probability of the discriminator for the real image x, D(G(z)) is the output probability of the discriminator for the image G(z) generated by the generator, E represents the expectation, and λ is the weight number used to balance the importance of the hinge loss and the gradient penalty term. represents the random interpolation between the real sample and the generated sample, Denotes the discriminator's random interpolation The gradient of

[0109] Hinge loss function L d Find the maximum possibility and expectation for the probability of the real data D(x) output by the discriminator and the image D(G(z)) generated by the discriminator output respectively, and sum the expectations of these two parts to get the hinge loss function part.

[0110] When training the discriminator, the parameters of the discriminator are continuously updated through optimization methods such as back-propagation algorithm and gradient descent to minimize the hinge loss function, that is, to enable the discriminator to accurately distinguish between real images and generated images.

[0111] For example, for a real image x, the discriminator should output a high probability (close to 1), so max(0,1-D(x)) encourages D(x) to approach 1.

[0112] For the generated image G(z), the discriminator should output a low probability (close to 0), so max(0,1+D(G(z))) encourages D(G(z)) to approach -1.

[0113] The gradient penalty term GP is used to prevent the gradient disappearance and gradient explosion problems, making the training process more stable. This is achieved by taking the L2 norm of the gradient and making it approach 1.

[0114] Specifically, represents the L2 norm of the gradient of the discriminator with respect to the random interpolation, Represents the square of the difference between the gradient norm and 1. By minimizing this penalty term, the gradient norm can be encouraged to remain near 1, thereby avoiding the gradient vanishing and gradient exploding problems.

[0115] In summary, the improved hinge loss function not only optimizes the performance of the discriminator but also stabilizes the training process by combining the hinge loss and the gradient penalty term, thereby improving the overall performance of the generative adversarial network.

[0116] In some embodiments, the loss function L of the generator is G By the mean square error loss L p , Perceptual loss L adv and content loss L c The composition is expressed as

[0117] L G =λ p *L p +λ c *L c +λ adv *L adv ,

[0118] Among them, λ p , c , adv They represent the mean square error loss L p , perceptual loss λ adv and content loss L c The weight coefficient of .

[0119] When training the generator, the goal is to maximize the discriminator's loss function, that is, to make the images generated by the generator deceive the discriminator as much as possible.

[0120] Through optimization methods such as back-propagation algorithm and gradient descent, the parameters of the generator are continuously updated to maximize the loss function of the discriminator.

[0121] According to the conclusion drawn by trial and error, when setting L p =0.5, L c 0.006, L adv When is 0.01, the discriminator training effect is best.

[0122] In some embodiments, the improved generative adversarial network is trained using a data set, and optimized multiple times by iteration until convergence to obtain a target generative adversarial network, including:

[0123] S4.1. Select a public dataset for the image deblurring task, including blurred images and corresponding real images; commonly used datasets include GoPro and RealBlur, which provide a large number of blurred and clear image pairs, suitable for training and testing image deblurring algorithms.

[0124] S4.2. Fix the generator parameters, input the blurred image into the generator, and generate the deblurred image.

[0125] S4.3. Input the generated deblurred image and the real image into the discriminator together, calculate the discriminator loss function, and let the discriminator distinguish between the two types of images.

[0126] S4.4. Use an optimizer to update the parameters of the discriminator to minimize the discriminator loss function and improve its ability to distinguish between real images and generated images; the optimizer can be, for example, the Adam optimizer.

[0127] S4.5. Fix the discriminator parameters and input the blurred image into the generator again to generate a new deblurred image.

[0128] S4.6. Calculate the generator loss function and use the optimizer to update the parameters of the generator to minimize the generator loss function.

[0129] S4.7. Repeat the iteration until the loss function converges to a predetermined value or reaches a preset number of iterations to obtain the target network.

[0130] Through the above steps, the improved generative adversarial network is trained using a public dataset, and multiple iterations are optimized until convergence, and finally a target generative adversarial network is obtained. The target generative adversarial network can effectively deblur the blurred image and generate high-quality clear images.

[0131] According to a third aspect of an embodiment of the present application, a method for image deblurring based on a generative adversarial network is provided, wherein a blurred image is input into the target network and a corresponding clear image is output.

[0132] The fourth aspect of the embodiment of the present application provides an image deblurring system, which uses Windows 11 as the operating system; Python 3.9 as the programming language; Pytorch as the deep learning framework to write the model, including data loading, model building, loss function calculation and optimizer update; NVIDIA RTX 3090 as the hardware platform, Adam as the optimizer, and trains the model until the model converges or reaches a predetermined number of training rounds.

[0133] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned method steps are implemented.

[0134] According to a sixth aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method steps when executing the computer program.

[0135] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for building a generative adversarial network, characterized in that: The following steps are involved: The basic architecture based on the Generative Adversarial Network (GAN) is adopted. The generator adopts the FPN feature pyramid structure, and the discriminator adopts the CNN convolutional neural network structure. In each layer of the FPN structure, a self-attention module is added to obtain an improved generative adversarial network; Among them, the self-attention module receives the feature map from FPN as input and converts it into the query matrix Q, key-value matrix K and value matrix V in the matrix space. The weight distribution of V is determined by calculating the similarity between Q and K, and the output of the self-attention feature vector is obtained.

2. A method for building a generative adversarial network according to claim 1, characterized in that: The fusion process of the self-attention module includes: The generator inputs a three-channel blurred image, and the shape of the feature map X output by each layer in the bottom-up path of FPN is Where C is the number of channels of the input image, and N is the set of w*h matrices of the input image; By querying the weight matrix w q , key value weight matrix w k , value weight matrix w v , the input feature map X is converted into the query matrix Q, key value matrix K and value matrix V through linear transformation, the expression formula is as follows [Q K V]=[w q w k w v ]X; Use Q and K to determine the weight distribution of V and get the self-attention feature vector output O, which is expressed as follows Among them, QK T is the dot product of the query matrix Q and the key matrix K, d q is the feature dimension of the query matrix Q, is the scaling factor, and the softmax function is used to convert the similarity score into a probability distribution, i.e., a weight distribution; Normalize the output self-attention feature vector O; At each layer of FPN, the output feature map is fused with the self-attention feature vector to enhance FPN’s capture of global information and obtain more detailed information.

3. The method for building a generative adversarial network according to claim 2, characterized in that: Query weight matrix w q , key value weight matrix w k , value weight matrix w v The dimensions are Among them, C is the input feature dimension, is the output feature dimension.

4. A training method for generating an adversarial network, characterized in that: include: Set the loss function of the generator and discriminator; The improved generative adversarial network described in any one of claims 1 to 3 is trained using a data set, and optimized multiple times until convergence to obtain a target network.

5. A training method for generating an adversarial network according to claim 4, characterized in that: The loss function of the discriminator adopts the improved hinge loss function L D , given by the hinge loss function L d And the gradient penalty term GP, its expression formula is L D =L d +λGP, Where D(x) is the output probability of the discriminator for the real image x, D(G(z)) is the output probability of the discriminator for the image G(z) generated by the generator, E represents the expectation, and λ is the weight number; represents the random interpolation between the real sample and the generated sample, Denotes the discriminator's random interpolation The gradient of Hinge loss function L d Find the maximum probability and expectation of the probability of the real data D(x) output by the discriminator and the image D(G(z)) generated by the discriminator output, and sum their expectations to optimize the performance of the discriminator; The gradient penalty term GP is used to prevent gradient disappearance and gradient explosion problems and make the training process more stable.

6. A training method for generating adversarial networks according to claim 4, characterized in that: The loss function L of the generator G By the mean square error loss L p , Perceptual loss L adv and content loss L c The composition is expressed as L G =λ p *L p +λ c *L c +λ adv *L adv , Among them, λ p , c , adv They represent the mean square error loss L p , Perceptual loss L adv and content loss L c The weight coefficient of .

7. A training method for generating adversarial networks according to claim 4, characterized in that: The improved generative adversarial network is trained using the dataset and optimized iteratively for multiple times until convergence to obtain the target generative adversarial network, including: Select a public dataset for the image deblurring task, including blurred images and corresponding real images; Fixed the generator parameters, input the blurred image into the generator, and generated the deblurred image; The generated deblurred image and the real image are input into the discriminator together and the discriminator loss function is calculated; Use the optimizer to update the parameters of the discriminator to minimize the discriminator loss function; Fix the discriminator parameters and input the blurred image into the generator again to generate a new deblurred image. Calculate the generator loss function and use the optimizer to update the generator parameters to minimize the generator loss function; Repeat the iteration until the loss function converges to a predetermined value or reaches a preset number of iterations to obtain the target network.

8. An image deblurring method based on a generative adversarial network, characterized in that: Input the blurred image into the target network described in any one of claims 4 to 7, and output the corresponding clear image.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the method steps of any one of claims 1 to 3 or 4 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method steps described in any one of claims 1 to 3 or 4 to 7 are implemented.

Images