Generative Adversarial Network Image Deblurring Method Based on Nonlinear Non-Activated Network

Abstract

The present invention discloses an image deblurring method for generative adversarial networks based on a non-linear activation-free network, aiming to efficiently process large-scale image deblurring problems. This method uses a non-linear activation-free network (NAFNet) as the generator network of the generative adversarial network. The generator network avoids traditional activation functions. In particular, NAFNet combines depthwise separable convolutions, simplified channel attention, and simple gate structures to effectively learn the complex features and structural information of images. In addition, the present invention also adopts a global and local dual-scale discrimination mechanism, which can more accurately capture the details and overall structure of images. The present invention also incorporates perceptual loss and adversarial loss to ensure that the generated images are visually closer to real images. The present invention provides an efficient and high-quality image deblurring method, which can achieve better performance than existing methods on multiple public datasets.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision and deep learning, and particularly relates to a generative adversarial network image deblurring method based on a non-linear activation-free network for blind image deblurring using a generative adversarial network (GAN). Background Art

[0002] Traditional image deblurring methods usually rely on complex optimization processes, which can be very time-consuming when dealing with large images. In recent years, deep learning techniques, especially adversarial networks (GANs), have been proven to perform excellently in various image restoration tasks. However, most existing networks use activation functions, such as ReLU, to introduce non-linearity, which may lead to unstable training and generate artifacts.

[0003] The background art involves: a non-linear activation-free network generator and a dual-scale discriminator

[0004] (1) Non-linear activation-free network generator

[0005] The non-linear activation-free network generator is a special deep neural network that does not rely on traditional activation functions to introduce non-linearity. Instead, it achieves non-linearity through a specially designed network structure and components. This design allows the network to perform effective forward and backward propagation without an activation function, thus improving the stability of training. In addition, due to the absence of an activation function, the network can reduce the generated artifacts and improve the quality of the generated images.

[0006] (2) Dual-scale discriminator

[0007] The dual-scale discriminator is a key component in the adversarial network. It contains two discriminators with different scales: one for evaluating local texture and details, and the other for evaluating global structure and content. This design ensures that the generated image is similar to the real image not only in local texture but also in global structure. The dual-scale discriminator provides more refined feedback to the generator, enabling it to generate higher-quality deblurred images. Summary of the Invention

[0008] The object of the present invention is to provide a generative adversarial network image deblurring method based on a non-linear activation-free network. This method uses a generator network with a specific structure to achieve non-linearity through specially designed network blocks without using an activation function. In addition, this method also employs a dual-scale discriminator to further improve the deblurring effect.

[0009] The specific technical solution for achieving the object of the present invention is as follows:

[0010] A method for deblurring images of generative adversarial networks based on a non-linear activation-free network, characterized in that the method comprises the following steps:

[0011] Step 1: Obtain an image training dataset

[0012] The dataset is divided into a blurred image dataset blur and a sharp image dataset sharp. These two datasets are paired, that is, each blurred image has a corresponding sharp image;

[0013] Step 2: Preprocess the dataset

[0014] First, preload the dataset, and then adjust the size of each pair of images to maintain the aspect ratio of the image, ensuring that the minimum side of the image is at least as large as the set parameter size, defaulting to 256x256; after image size adjustment, perform normalization and transposition, and transpose it from the height * width * channel format to the channel * height * width format; then enhance the data, namely rotate, crop and flip; finally, intercept 10% of the dataset randomly as the validation set, and the rest are the training sets;

[0015] Step 3: Initialize the loss functions of the generator and the discriminator

[0016] The generator loss L G includes the perceptual loss L perceptual , the pixel-level loss L pixel and the adversarial loss L adversarial ; the discriminator loss L D adopts the RaGAN-LS loss function;

[0017] Step 4: Construct the network models of the generator G and the discriminator D, and iteratively optimize the parameters of the generator G and the discriminator D until the losses of all networks converge

[0018] The network model of the generator G is NAFNet. The NAFNet structure consists of several non-linear blocks NAFBlock. The neural network components of the non-linear block NAFBlock are depthwise separable convolutional layers, channel attention layers, simple gating layers and feed-forward network layers;

[0019] The discriminator D uses a dual-scale discriminator, which consists of a local discriminator PatchGAN and a global discriminator FullGAN. The local discriminator PatchGAN captures the local features of the image through a 12-layer neural network structure to ensure that the generated image matches the real image in terms of details and textures. The global discriminator FullGAN captures the context of the image through an 18-layer neural network structure to ensure that the generated image is similar to the real image in terms of overall structure and layout. The network layer components of both the global discriminator FullGAN and the local discriminator PatchGAN are composed of depthwise separable convolutional layers, layer normalization layers, and GELU layers, except that their respective numbers of layers are different. The global discriminator FullGAN is responsible for receiving all image information, and the local discriminator PatchGAN is responsible for receiving the image information after random cropping.

[0020] Use the blur images in the training set of step 2 as the input images and the sharp images as the target images. Use the generator G to generate output images, and input the output images and the target images into the discriminator to calculate the discriminator loss L. D And update the discriminator weights; clear the gradients of the generator; calculate the generator loss L. G = pixel-level loss L pixel + perceptual loss L perceptual + adversarial loss L adversarial , backpropagate the gradients of the generator and update the weights of the generator; the generator G and the discriminator D will be updated alternately. The generator G tries to generate more and more realistic images, while the discriminator D tries to distinguish between real images and generated images. This mutually adversarial relationship continuously drives both forward until a balance is reached, that is, the images generated by the generator are so realistic that it is difficult for the discriminator to distinguish between generated images and real images.

[0021] During the model training process, the Adam optimizer is used as the model optimization method to dynamically adjust the learning rate of each parameter; the generator loss L is continuously optimized through the adversarial training model optimization method. G and the discriminator loss L D , until the loss converges.

[0022] Step 5: When the model training is completed, save the weights and structure of the trained model. Input the preprocessed image data into the trained network model, perform data forward propagation calculation, and then perform post-processing on the output format conversion of the deblurred image data output by the network model, and finally return the clear image after deblurring.

[0023] The said step 1 specifically includes:

[0024] Step a1: Capture blurred images and clear images through professional equipment, or use existing public datasets such as the GoPro dataset and the Kohler dataset as the original dataset;

[0025] Step a2: Divide the dataset into test and train datasets, with the train dataset accounting for 90%; under each dataset, the blurred images and the corresponding clear images are respectively divided into two sets, namely the blur set and the sharp set. The images in the blur set correspond one-to-one with the images in the sharp set, which is simply called an image pair.

[0026] The specific steps of step 2 include:

[0027] (1) Data selection and sampling

[0028] Select a subset from the original dataset, select based on the hash value of the data, and divide the training set and the validation set in a 9:1 ratio;

[0029] (2) Loading data

[0030] After reading the images, load all the images into memory in a pre-loading manner; the entire dataset only needs to be read once, and all subsequent accesses are directly from memory, thus greatly accelerating the data reading speed;

[0031] (3) Image cropping and enhancement

[0032] Cropping: Randomly select a starting point (x, y), and use this point as the upper-left corner coordinate of the cropping area to crop an image of size 256*256. Ensure that the cropped image is within the boundaries of the original image to avoid any possible boundary problems;

[0033] Enhancement: Flip the image along its horizontal axis with a 50% probability, translate, scale, and rotate the image according to random parameters, flip the image along its main diagonal with a 50% probability, simulate the optical distortion effects of fisheye, barrel, or pillow distortion, and apply a non-rigid transformation to the image to simulate the distortion of natural forms;

[0034] (4) Image normalization

[0035] Convert the image from the height*width*channel format to the channel*height*width format, and normalize the image, converting the pixel values from the range [0, 255] to the range [0, 1].

[0036] Step 2 provides a complete data preprocessing framework, including data selection, loading, resizing, cropping, enhancement, normalization, and optional image corruption. These preprocessing steps ensure that the model receives appropriate and diverse data during the training process, thereby improving its performance and generalization ability.

[0037] Step 3 specifically includes:

[0038] Define the perceptual loss L perceptual , the pixel-level loss L pixel and the adversarial loss L adversarial to jointly act on the loss L of the generation network G ; The perceptual loss L perceptual represents the difference between the generated image and the real image in a certain feature space; the pixel-level loss L pixel represents the difference between the generated image and the real image at the pixel level; the adversarial loss L adversarial represents the ability of the generator to try to deceive the discriminator; the discriminator loss L D uses the RaGAN-LS loss, which is the difference between the prediction result of the real image in the discriminator D and the average prediction of the forged image, and the difference between the prediction result of the forged image in the discriminator D and the average prediction of the real image. The loss function of the discriminator D is to calculate the average of the two differences.

[0039] Step 4 specifically includes:

[0040] The network model of the generator in Step 4 is NAFNet. A complete U-Net structure formed by combining an encoder Encoder and a decoder Decoder is designed in the generator. Each layer of the encoder is connected to the corresponding layer of the decoder through skip connections, enabling the generator to capture detailed information at a deep level. Finally, the output of the decoder passes through a Sigmoid activation function, making the output value within the range of [0,1]; where the encoder consists of 5 non-linear blocks NAFBlock, and the number of channels of each non-linear block is increasing, and its function is to gradually downsample the input image and extract its features; the decoder consists of 4 non-linear blocks NAFBlock and a convolutional layer, and its function is to gradually upsample the features and finally restore to the original size of the input image.

[0041] The local discriminator PatchGAN described in step 4 captures the local features of the image through its 12-layer network structure, ensuring that the generated image matches the real image in terms of details and textures; the global discriminator FullGAN captures the context of the image through its 18-layer network structure, ensuring that the generated image is similar to the real image in terms of overall structure and layout; both discriminators are composed of depthwise separable convolutional layers, layer normalization layers, and GELU activation layers; layer normalization normalizes in the channel direction, making the training smoother and more efficient; the Gaussian error linear unit GELU activation function can provide a smoother gradient calculation, which helps with parameter optimization and performs better in terms of training stability and model robustness; during the training process, the global discriminator FullGAN is responsible for receiving all image information, and the local discriminator PatchGAN is responsible for receiving the local image information after random cropping; the losses of the two discriminators are combined to generate an image with both good local details and a suitable global structure.

[0042] The model optimization method for adversarial training described in step 4 uses the gradient descent algorithm based on the Adam optimizer to iteratively optimize the parameters of the discriminator D and the generator G, which specifically includes the following steps:

[0043] (1) Initialize the weight parameters of the discriminator D and the generator G; set the generator and the discriminator to the training mode, which means that all layers in the model (such as Dropout and Layer Normalization) will run in the training mode, set the number of iterations epoch to 0, and set the maximum number of iterations num_epochs;

[0044] (2) Iterate through the data loader

[0045] For each batch of data in the data loader, perform the following operations:

[0046] ⅰ) Input the blurred image into the non-linear non-activation network generator G to generate a de-blurred image;

[0047] ⅱ) Use the dual-scale discriminator D to evaluate the authenticity of the generated de-blurred image;

[0048] ⅲ) Calculate the generator loss L G and the discriminator loss L D , including the pixel-level loss L pixel the perceptual loss L perceptual and the adversarial loss L adversarial ;

[0049] ⅳ) Use the optimizer to update the weights of the generator D and the discriminator G, use the learning rate scheduler to update the learning rates of the generator G and the discriminator D, and increment epoch by one;

[0050] v) Repeat steps i)-iv) to converge all losses until the training ends at the preset maximum number of iterations.

[0051] The present invention adopts a non-linear non-activation network structure as the generator, the core of which is NAFNet. This design allows the model to achieve a high degree of non-linearity without traditional activation functions. In terms of the discriminator, the present invention adopts the design of a dual-scale discriminator, specifically including a local discriminator PatchGAN and a global discriminator Full-GAN. Among them, the depth of PatchGAN is 12 layers, and the depth of Full-GAN is 18 layers, and the normalization layer and activation layer in the network layer are optimized. This dual-scale structure ensures that real and generated images can be effectively recognized at different scales.

[0052] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0053] 1. Higher non-linear expression ability: By using a non-linear non-activation network generator, the present invention achieves a powerful non-linear modeling ability without using conventional activation functions. This enables the network to maintain the ability to accurately capture image details while avoiding training instability and generated artifacts caused by activation functions.

[0054] 2. Efficient image deblurring: Combining the improved dual-scale discriminator and a specific loss function, the present invention provides a powerful and efficient deblurring solution, which can more effectively process various blurred images and generate clear and detail-rich results.

[0055] 3. Better perceptual quality: By combining pixel-level loss, perceptual loss, and adversarial loss, the present invention can produce deblurred results that are perceptually closer to real images. The perceptual loss ensures that at a specific feature level, the generated image is similar to the real image in terms of structure and texture.

[0056] 4. More stable training: The network design and loss function selection of the present invention are aimed at ensuring the stability of the training process, thereby reducing the training difficulty and the required time.

[0057] 5. Wider application range: Although the present invention is mainly aimed at image deblurring, its core ideas and methods can also be applied to other image restoration tasks, such as denoising, super-resolution, and color correction.

[0058] 6. Reduced computational requirements: Since the use of a large number of activation functions is avoided, the model of the present invention may require less computational resources and time for forward and backward propagation in some cases. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flowchart of the present invention;

[0060] Figure 2 Schematic diagram of the adversarial network model of the present invention;

[0061] Figure 3 Network structure diagram of the NAFBlock, the non - linear block of NAFNet in the present invention;

[0062] Figure 4 Schematic diagram of the local discriminator network model in the dual - scale discriminator of the present invention;

[0063] Figure 5 Schematic diagram of the global discriminator network model in the dual - scale discriminator of the present invention. Detailed implementation manners

[0064] The following further elaborates on the present invention in conjunction with specific embodiments and the accompanying drawings. The processes, conditions, experimental methods, etc. for implementing the present invention, except for the specifically mentioned content below, are all common knowledge and well - known general knowledge in the art, and the present invention has no particularly restricted content.

[0065] The present invention includes:

[0066] Step 1) Obtain an image training dataset. The dataset is divided into a blurred image dataset blur and a sharp image dataset sharp. These two datasets are paired, that is, each blurred image has a corresponding sharp image.

[0067] Step 2) Pre - process the dataset. First, pre - load the dataset, and then adjust the size of each pair of images to maintain the aspect ratio of the image, ensuring that the minimum side of the image is at least as large as the set parameter "preload_size", generally defaulting to 256x256. After adjusting the image size, perform normalization and transposition, and transpose it from the HxWxC (height x width x channels) format to the CxHxW format. Then enhance the data, such as rotation, cropping, flipping, etc. Finally, intercept a part of the dataset as the validation set, and the rest are the training sets.

[0068] Step 3) Initialize the loss function L of the generator G and the loss function L of the discriminator D . Use perceptual loss for the generator and RaGAN - LS loss for the discriminator.

[0069] Step 4) Construct the network models of the generator and the discriminator. The network model of the generator is NAFNet, and its main structure is NAFBlock. The main components of NAFBlock are depthwise separable convolution, simplified channel attention, SimpleGate, and feed-forward network, aiming to achieve efficient image generation with a simple network. The discriminator adopts a dual-scale discriminator, namely the local discriminator (PatchGAN) and the global discriminator (FullGAN). The former usually focuses on the local regions or "patches" of the image and usually has a smaller receptive field, enabling it to focus on the details and textures of the image. The goal of the local discriminator is to determine whether each local region comes from a real image or a generated image. The global discriminator evaluates the entire image, capturing the global structure and context information of the image, ensuring that the generated image is globally similar to the real image. The motivation for this dual-scale discriminator approach is to capture the local and global features of the image, thus better distinguishing real images from generated images.

[0070] Step 5) Model training. Use the blur images in the training set in Step 2 above as the input images and the sharp images as the target images. Use the generator to generate output images, and input the output images and the target images into the discriminator together to calculate the discriminator loss and update the discriminator weights; clear the gradients of the generator; calculate the total loss L of the generator G = pixel-level loss L pixel + perceptual loss L perceptual + adversarial loss L adversarial , backpropagate the gradients of the generator and update the weights of the generator. The generator loss function and the discriminator loss function are updated alternately. The generator tries to generate more and more realistic images, while the discriminator tries to distinguish real images from generated images. This adversarial relationship continuously drives both forward until a balance is reached, that is, the images generated by the generator are so realistic that the discriminator has difficulty distinguishing them from real images.

[0071] After the model training is completed, evaluate the performance of the model on the validation dataset to ensure that the model not only performs well on the training data but also can make accurate predictions on unseen data.

[0072] Step 6) After the model training is completed, save the weights and structure of the trained model. Given any blurred image, call the model to obtain the corresponding sharp image.

[0073] The specific content of Step 1) includes:

[0074] Step a1: Capture blurred images and sharp images through professional equipment, or use existing datasets, such as the GoPro dataset, as the original dataset;

[0075] Step a2: Divide the dataset into test and train datasets, with the train dataset accounting for the majority. Under each dataset, the blurred images and their corresponding clear images are respectively divided into two sets, namely the blur set and the sharp set. The images in the blur set all correspond one-to-one with the images in the sharp set, which is simply referred to as an image pair.

[0076] The specific steps of step 2) include:

[0077] (1) Data selection and sampling

[0078] Select a subset from the original dataset. It is selected based on the hash value of the data, and the training set and validation set are divided in a 9:1 ratio.

[0079] (2) Loading data

[0080] After reading the images, all images are loaded into memory in a pre-loading manner. The entire dataset only needs to be read once, and all subsequent accesses are directly from memory, thus greatly accelerating the data reading speed.

[0081] (3) Image cropping and enhancement

[0082] Cropping: Randomly select a starting point (x, y), and use this point as the upper left corner coordinate of the cropping area to crop an image of size 256*256. Ensure that the cropped image is within the boundaries of the original image to avoid any possible boundary problems.

[0083] Enhancement: Flip the image along its horizontal axis with a 50% probability, translate, scale, and rotate the image according to random parameters, flip the image along its main diagonal with a 50% probability, simulate optical distortion effects such as fisheye, barrel, or pillow distortion, and apply non-rigid transformations to the image to simulate the distortion of natural forms.

[0084] (4) Image normalization

[0085] Convert the image from the (height, width, channels) format to the (channels, height, width) format, and normalize the image, converting the pixel values from the range [0, 255] to the range [0, 1].

[0086] (5) Image corruption processing

[0087] If an image corruption function is defined, it will be applied to the input image to simulate various possible corruptions or noises.

[0088] This step provides a complete data preprocessing framework, including data selection, loading, resizing, cropping, augmentation, normalization, and optional image corruption. These preprocessing steps ensure that the model receives appropriate and diverse data during training, thereby improving its performance and generalization ability.

[0089] The specific content of step 3) is as follows: Define a content-aware (perceptual) loss for the generation network. The perceptual loss helps to preserve the structure and content of the image while allowing for minor pixel-level differences. The discriminator uses the RaGAN-LS loss. The goal of RaGAN-LS is to make the discriminator output for real images close to the average of the discriminator outputs for fake images plus a certain constant, while the discriminator output for fake images is close to the average of the discriminator outputs for real images minus a certain constant.

[0090] The specific content of step 4) is as follows: The generator designs a complete U-Net structure formed by combining an encoder and a decoder based on NAFNet. Each layer of the encoder is connected to the corresponding layer of the decoder through skip connections, which helps the network capture detailed information at a deep level. Finally, the output of the decoder passes through a Sigmoid activation function, making the output value within the range of [0, 1]. The encoder consists of 5 non-linear blocks, NAFBlock. Among them, the first three non-linear blocks each contain one block, the fourth non-linear block contains 18 blocks; the fifth non-linear block contains one block, which serves as an intermediate block connecting the encoder and the decoder, and its role is to further process the output features of the encoder. The number of channels in each non-linear block increases, and its role is to gradually downsample the input image and extract its features. The decoder consists of 4 non-linear blocks and a convolutional layer, and its role is to gradually upsample the features and finally restore to the original size of the input image.

[0091] NAFNet adopts a special network design that can introduce non-linear characteristics even without activation functions. This is mainly achieved in the following ways:

[0092] (1) Non-linear block (NAFBlock): In the non-linear block, multiple convolutional operations are stacked together without any activation function intervening. A single convolutional operation is linear, but the continuous stacking of multiple convolutional operations can introduce non-linearity. This is because the output of a convolutional operation can be used as the input of the next convolutional operation, so that the effects of multiple layers of convolution can be superimposed on each other.

[0093] (2) Skip Connection: NAFNet uses skip connections that can span certain layers of the network. These skip connections can not only help the network capture more information in the deeper layers but also strengthen the network's non-linearity. By combining information from different layers, skip connections provide a mechanism for the network to learn more complex representations without activation functions.

[0094] (3) Multi-scale Features: Due to its U-Net structure, NAFNet can capture features at different scales. This means the network can consider both high-level and low-level image features simultaneously. This multi-scale property enhances the network's non-linearity and helps it better handle various image tasks.

[0095] The generator introduces non-linearity without activation functions through the unique design of NAFNet. This design can reduce the computational complexity in the network while still maintaining strong representation capabilities.

[0096] The discriminator in step 4) uses a dual-scale discriminator method, a local image discriminator (PatchGAN) and a global image discriminator (FullGAN). PatchGAN captures local features of the image through its shallower (12 layers) network structure, ensuring that the generated image matches the real image in terms of details and textures. FullGAN captures a larger range of context of the image through its deeper (18 layers) network structure, ensuring that the generated image is similar to the real image in terms of overall structure and layout. During training, the losses of the two discriminators are combined. This encourages the generator to generate images that have both good local details and a suitable global structure. In this way, the design of the dual-scale discriminator ensures that the generator produces realistic images not only in local details but also in global structure. This method provides a more comprehensive way to evaluate the quality of the generated image, making it more difficult for the generator to deceive the discriminator and thus producing higher-quality outputs.

[0097] The model optimization method based on adversarial training in step 5) uses a gradient descent algorithm based on the Adam optimizer to iteratively optimize the parameters of the discriminator D and the generator G, specifically including the following steps:

[0098] (1) Initialize the weight parameters of the discriminator D and the generator G; set the generator and the discriminator to training mode, which means all layers in the model (such as Dropout and Batch Normalization) will run in training mode, set the number of iterations epoch to 0, and set the maximum number of iterations num_epochs;

[0099] (2) Iterate through the data loader

[0100] For each batch of data in the data loader, perform the following operations:

[0101] 1. Input the blurred image into the non-linear activation-free network generator G to generate a de-blurred image.

[0102] 2. Use the dual-scale discriminator D to evaluate the authenticity of the generated de-blurred image.

[0103] 3. Calculate the generator loss L G and the discriminator loss L D , including the pixel-level loss L pixel , the perceptual loss L perceptual and the adversarial loss L adversarial .

[0104] 4. Use the optimizer to update the weights of the generator D and the discriminator G, use the learning rate scheduler to update the learning rates of the generator G and the discriminator D, and increment the epoch by one.

[0105] 5. Repeat the above steps until all losses converge and the training ends when the preset maximum number of iterations is reached.

[0106] Embodiment

[0107] The entire process of this embodiment includes steps 1)-5), please refer to Figure 1 and Figure 2 . Figure 1 is the flowchart of the generative adversarial network image de-blurring method based on the non-linear activation-free network in the embodiment, Figure 2 is the schematic diagram of the generative adversarial network model based on the non-linear activation-free network.

[0108] The specific process of this embodiment:

[0109] Step 1) Obtain the image training dataset. The dataset is divided into a blurred image dataset blur and a sharp image dataset sharp. These two datasets are paired, that is, each blurred image has a corresponding sharp image.

[0110] Step 2) Preprocess the dataset. The preprocessed blur image is set as the input image input of the generator, and the sharp image is set as the target of the discriminator.

[0111] Step 3) Initialize the loss functions of the generator and the discriminator. The generator loss usually involves multiple parts, including perceptual loss, pixel-level loss, and adversarial loss. The discriminator mainly uses the RaGAN-LS loss.

[0112] The generator loss is as follows:

[0113] L G= λ1L pixel + λ2L perceptual + λ3L adversarial

[0114] where L pixel is the pixel - level loss, usually the absolute value (L1 loss) or Euclidean distance (L2 loss) between the generated image and the real image. The pixel - level loss function can correct color and texture distortions; L perceptual is the perceptual loss, the feature distance calculated based on specific layers of a pre - trained network (such as VGG19); L adversarial is the adversarial loss, measuring the score of the generated image in the discriminator. λ1, λ2, λ3 are hyperparameters that weigh the importance of different loss terms. The adversarial loss L adversarial has the same calculation formula as the discriminator loss L D , both are dual - scale discriminant calculations. In this embodiment, the hyperparameter settings are λ1 = 0.5, λ2 = 0.006, λ3 = 0.01, and the actual formula for calculating the generator loss L G is calculated as follows:

[0115] L G = 0.5 * L pixel + 0.006 * L perceptual + 0.01 * L perceptual

[0116] The discriminator loss L D is an upgrade based on the WGAN - GP loss, a dual - scale discriminator loss RaLSGAN composed of global and local scales, and the generated results have higher perceptual quality and clearer overall output. The formula for calculating the discriminator loss L D is calculated as follows:

[0117]

[0118] where, is the expectation of the real data distribution; is the expectation of the image distribution generated by the generator; D(x) is the output of the discriminator D for the real image x; D(G(z)) is the output of the discriminator for the generated image G(z). This loss function contains two main parts: The first part is to calculate the difference between the score of the real image in the discriminator D and the average score of the generated image in the discriminator D, with the goal of ensuring that the score of the real image is higher than the average score of the generated image; The second part is to calculate the difference between the score of the generated image in the discriminator D and the average score of the real image in the discriminator D, with the goal of ensuring that the score of the generated image is lower than the average score of the real image. Combining these two parts ensures that the discriminator D can correctly distinguish between real images and generated images, and at the same time provides feedback to the generator G on how to improve its output.

[0119] The RaGAN-LS loss provides a more stable and effective method for determining real and fake images by considering the relative scores of real images and generated images in the discriminator, which helps the generator produce higher-quality generated images.

[0120] Step 4) Construct the network models of generator G and discriminator D

[0121] The network model of the generator is NAFNet, see Figure 2 In the left part, the main substructure of NAFNet is NAFBlock, see Figure 3 , and the main components of NAFBlock are depthwise separable convolutional layer, channel attention layer, simple gating layer, and feed-forward network layer, aiming to achieve efficient image generation with a simple network. In this embodiment, the distribution of the blocks included in the non-linear block NAFBlock is [1, 1, 1, 18]*1*[1, 1, 1, 1]. Through experiments, a simple distribution of NAFBlock will lead to a too weak generator, and if it is too complex, the calculation time will be very long. This distribution method is the optimal method of the current experimental results.

[0122] · Depthwise Separable Convolution:

[0123] This is a special convolution operation, which can be regarded as a lightweight version of the standard convolution. It first performs separate spatial convolutions on each channel of the input, and then uses 1*1 convolution (also called pointwise convolution) to combine these channels. Compared with the standard convolution, it can greatly reduce the number of model parameters and the amount of calculation.

[0124] ● Simplified Channel Attention:

[0125] Channel attention assigns different weights to different channels of the input feature map, so that the network can focus on more important channels. In the present invention, it uses global average pooling and global maximum pooling, and then combines the two to obtain channel attention weights through a fully connected layer.

[0126] ● Simple Gating Layer

[0127] The simple gating layer is a simple neural network module that divides the input features into two parts and then modulates them through element-wise multiplication. Among them, one part provides the original information, and the other part serves as the modulation signal, determining the strengthening or suppression of the original information. This provides a simple and efficient feature adjustment mechanism for the network.

[0128] ● Feed-Forward Network

[0129] This is a network composed of traditional two-layer fully connected layers, which is used in the NAFBlock to further extract features.

[0130] The discriminator adopts a dual-scale discriminator, namely the local discriminator PatchGAN and the global discriminator FullGAN. The network model of the local discriminator PatchGAN refers to Figure 4 , and the network model of the global discriminator FullGAN refers to Figure 5 ; the former focuses on the local regions of the image, and the latter focuses on the overall image. This discriminator usually has a relatively small receptive field, enabling it to focus on the details and textures of the image. The goal of the local discriminator is to determine whether each local region comes from a real image or a generated image. The global discriminator evaluates the entire image, capturing the global structure and context information of the image, ensuring that the generated image is similar to the real image as a whole. The global discriminator has a total of 18 layers, processing the entire input image; the local discriminator has 12 layers, operating at the image patch level. Such a design ensures that the generator produces realistic images not only in local details but also in global structure. This approach provides a more comprehensive way to evaluate the quality of the generated image, making it more difficult for the generator to deceive the discriminator, thus producing higher-quality outputs.

[0131] Step 5) Model training.

[0132] Load the dataset. For each epoch and for each batch in the dataset, input the input into the generator. Through the generator G and the random noise z from the noise distribution, generate the fake image output; input the generated fake image output and the real image target into the discriminator for discrimination, and calculate the discriminator D loss L D ; update the weights of the discriminator according to the loss of the discriminator; use the generated fake image and the real image to calculate the loss L G of the generator G, which includes the pixel-level loss L pixel , the perceptual loss L perceptual and the adversarial loss L adversarial ; update the weights of the generator according to the loss of the generator.

[0133] Save the weights of the model every certain number of epochs. Record and / or print the loss and other relevant metrics every certain number of steps. The generated fake images may also be saved periodically to monitor the progress of the generator. Save the weights of the final model after the training is completed.

[0134] There are two objective functions:

[0135] 1. Optimize the discriminator loss function: Minimize L D ;

[0136] 2. Optimize the generator loss function: Minimize L G ;

[0137] In most GAN implementations, since optimization libraries (such as Adam or SGD) usually only perform minimization, the actual implementation may slightly adjust the objective function to ensure only minimization operations. This is also the case in this embodiment. Specifically, to maximize the formula, the objective function is usually negated and then minimized. In summary, although both losses are minimized in the method, the goals of these two losses are opposite: the discriminator tries to better distinguish real and generated images, while the generator tries to deceive the discriminator into thinking that the generated images are real.

[0138] The final objective function can be defined as:

[0139]

[0140] For the discriminator loss function L D :

[0141]

[0142] For the generator loss function L G :

[0143]

[0144] The core of the entire training process is adversarial. The losses of the generator loss function and the discriminator loss function are alternately optimized. The generator tries to produce as realistic images as possible to deceive the discriminator, while the discriminator's goal is to distinguish real and generated images as much as possible. This adversarial dynamics results in the generator generating more realistic images over time.

[0145] To accelerate the training of the network and obtain better performance, an adaptive learning rate adjustment technique is adopted. Specifically, the Cyclical Learning Rates strategy is used, which allows the learning rate to be dynamically adjusted during the training process. The main advantage of this strategy is that it can avoid the model getting stuck in local optima during training and can converge faster. By dynamically adjusting the learning rate at different stages of training, it is ensured that the model can quickly learn in the initial stage and fine-tune in the subsequent stages, thus avoiding overfitting.

[0146] Features of the present invention:

[0147] 1. Nonlinear non-activation network generator: A specially designed network block is adopted to achieve non-linearity without using an activation function. This can not only improve the training stability but also reduce the generated artifacts.

[0148] 2. Dual-scale discriminator: Two discriminators with different scales that are optimized are adopted. One focuses on local texture details and the other focuses on global structure. This ensures that the generated deblurred images have high quality at different scales.

[0149] 3. Perceptual loss: In addition to the traditional pixel-level loss, perceptual loss is also used to ensure that the generated images are visually closer to the real images.

[0150] The protection scope of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be conceived by those skilled in the art are included in the present invention, and the appended claims are taken as the protection scope.

Claims

1. A method for generating adversarial network image deblurring based on a non-linear non-activation network, characterized in that, The method includes the following steps: Step 1: Obtain an image training dataset The dataset is divided into a blurred image dataset blur and a sharp image dataset sharp. These two datasets are paired, that is, each blurred image has a corresponding sharp image; Step 2: Preprocess the dataset First, preload the dataset, and then adjust the size of each pair of images to maintain the aspect ratio of the images, ensuring that the minimum side of the image is at least as large as the set parameter size, which is 256x256 by default; after the image size adjustment, perform normalization and transposition, and transpose it from the height * width * channel format to the channel * height * width format; then enhance the data, namely rotation, cropping, and flipping; finally, intercept 10% of the dataset randomly as the validation set, and the rest are the training sets; Step 3: Initialize the loss functions of the generator G and the discriminator D Generator loss $L$ G includes perceptual loss $L$ perceptual , pixel-level loss $L$ pixel and adversarial loss $L$ adversarial ; Discriminator loss $L$ D adopts the RaGAN-LS loss function; Step 4: Construct the network models of the generator G and the discriminator D, and iteratively optimize the parameters of the generator G and the discriminator D until the losses of all networks converge The network model of the generator G is NAFNet. The NAFNet structure consists of several non-linear blocks NAFBlock. The neural network components of the non-linear block NAFBlock include a depthwise separable convolutional layer, a channel attention layer, a simple gating layer, and a feed-forward network layer; The discriminator D uses a dual-scale discriminator. The dual-scale discriminator consists of a local discriminator PatchGAN and a global discriminator FullGAN; PatchGAN focuses on the local regions of the image; PatchGAN has a receptive field that enables it to focus on the details and textures of the image; the goal of PatchGAN is to determine whether each local region comes from a real image or a generated image; FullGAN evaluates the entire image, capturing the global structure and context information of the image, making the generated image similar to the real image as a whole; The network layer components in the dual-scale discriminator include a depthwise separable convolutional layer, a layer normalization layer, and a GELU activation layer; Use the blur images in the training set of step 2 as the input images, the sharp images as the target images, use the generator G to generate the output images, and input the output images and the target images into the discriminator together to calculate the discriminator loss L D And update the discriminator weights; clear the gradients of the generator; Calculate the generator loss L G = Pixel-level loss L pixel + Perceptual loss L perceptual + Adversarial loss L adversarial , backpropagate the gradients of the generator and update the weights of the generator; The generator G and the discriminator D are updated alternately; the generator G tries to generate more and more realistic images, while the discriminator D tries to distinguish between real images and generated images; this adversarial relationship continuously drives both forward until a balance is reached, that is, the images generated by the generator are so realistic that it is difficult for the discriminator to distinguish between generated images and real images; During the model training process, the Adam optimizer is used as the model optimization method to dynamically adjust the learning rate of each parameter; the generator loss L and the discriminator loss L are continuously optimized through the adversarial training model optimization method until the loss converges; G and the discriminator loss L D , until the loss converges; Step 5: When the model training is completed, save the weights and structure of the trained model, input the preprocessed image data into the trained network model, perform data forward propagation calculation, and then perform post-processing on the deblurred image data output by the network model after output format conversion, and finally return the deblurred sharp image.

2. The method for generating and adversarial network image deblurring based on a non-linear non-activation network according to claim 1, wherein The specific content of Step 1 includes: Step a1: Capture blurred images and sharp images through professional equipment, or use existing public datasets such as the GoPro dataset and the Kohler dataset as the original dataset; Step a2: Divide the dataset into test and train datasets, with the train dataset accounting for 90%; under each dataset, divide the blurred images and the corresponding clear images into two sets, namely the blur set and the sharp set. The images in the blur set correspond one-to-one with the images in the sharp set, which are simply referred to as image pairs.

3. The method for generating adversarial network image deblurring based on a non-linear non-activation network as claimed in claim 1, wherein The specific steps of Step 2 include: (1) Data selection and sampling Select a subset from the original dataset based on the hash value of the data, and divide the training set and the validation set in a 9:1 ratio. (2) Loading data After reading the images, load all the images into memory in a pre-loading manner; the entire dataset only needs to be read once, and all subsequent accesses are directly from memory. (3) Image cropping and enhancement Cropping: Randomly select a starting point (x, y) and use this point as the upper left corner coordinate of the cropping area to crop an image of size 256*256, ensuring that the cropped image is within the boundaries of the original image. Enhancement: Flip the image along its horizontal axis with a probability of 50%, translate, scale, and rotate the image according to random parameters, flip the image along its main diagonal with a probability of 50%, simulate the optical distortion effects of fisheye, barrel, or pillow distortion, and apply a non-rigid transformation to the image to simulate the distortion of natural forms. (4) Image normalization Convert the image from the height*width*channel format to the channel*height*width format and normalize the image, converting the pixel values from the range [0, 255] to the range [0, 1].

4. The method for generating and adversarial network image deblurring based on a non-linear non-activation network according to claim 1, characterized in that, The specific steps of Step 3 include: Define the perceptual loss L perceptual , the pixel-level loss L pixel and the adversarial loss L adversarial act together on the loss L of the generation network G ; the perceptual loss L perceptual represents the difference between the generated image and the real image in a certain feature space; the pixel-level loss L pixel represents the difference between the generated image and the real image at the pixel level; the adversarial loss L adversarial represents the ability of the generator to try to deceive the discriminator; the discriminator loss L D uses the RaGAN-LS loss, which is the difference between the prediction result of the real image in the discriminator D and the average prediction of the forged images, and the difference between the prediction result of the forged images in the discriminator D and the average prediction of the real images. The loss function of the discriminator D is to calculate the average of the two differences.

5. The method for generating and adversarial network image deblurring based on a non-linear non-activation network according to claim 1, wherein The network model of the generator G in Step 4 is NAFNet. In the generator G, a complete U-Net structure is designed by combining an encoder Encoder and a decoder Decoder. Each layer of the encoder is connected to the corresponding layer of the decoder through skip connections, enabling the generator to capture detailed information at a deep level. Finally, the output of the decoder passes through a Sigmoid activation function to make the output value within the range [0, 1]; the encoder consists of 5 non-linear blocks NAFBlock, and the number of channels in each non-linear block increases. Its function is to gradually downsample the input image and extract its features; the decoder consists of 4 non-linear blocks NAFBlock and a convolutional layer, and its function is to gradually upsample the features and finally restore to the original size of the input image; the neural network components of the non-linear block NAFBlock include a depthwise separable convolutional layer, a channel attention layer, a simple gating layer, and a feed-forward network layer.

6. The method for generating and adversarial network image deblurring based on a non-linear non-activation network according to claim 1, wherein, The discriminator D in Step 4 uses a dual-scale discriminator, which consists of a local discriminator PatchGAN and a global discriminator FullGAN; the local discriminator PatchGAN captures the local features of the image through a 12-layer neural network structure to ensure that the generated image matches the real image in terms of details and textures; the global discriminator FullGAN captures the context of the image through an 18-layer neural network structure to ensure that the generated image is similar to the real image in terms of overall structure and layout. The network layer components of both the global discriminator FullGAN and the local discriminator PatchGAN are composed of depthwise separable convolutional layers, layer normalization layers, and GELU activation layers, only with different numbers of layers for each; the global discriminator FullGAN is responsible for receiving all image information, and the local discriminator PatchGAN is responsible for receiving the image information after random cropping.

7. The method for generating adversarial network image deblurring based on a non-linear non-activation network according to claim 1, wherein For the model optimization method of adversarial training described in step 4, the gradient descent algorithm based on the Adam optimizer is used to iteratively optimize the parameters of the discriminator D and the generator G, which specifically includes the following steps: (1) Initialize the weight parameters of the discriminator D and the generator G; set the discriminator D and the generator G to the training mode, which means that all layers in the model will run in the training mode, set the number of iterations epoch to 0, and set the maximum number of iterations num_epochs; (2) Iterate through the data loader For each batch of data in the data loader, perform the following operations: ⅰ) Input the blurred image into the non-linear non-activation network generator G to generate a deblurred image; ⅱ) Use the dual-scale discriminator D to evaluate the authenticity of the generated deblurred image; ⅲ) Calculate the generator loss L G and the discriminator loss L D , including the pixel-level loss L pixel perceptual loss L perceptual and adversarial loss L adversarial ; ⅳ) Update the weights of the generator D and the discriminator G using the optimizer, update the learning rates of the generator G and the discriminator D using the learning rate scheduler, and increment epoch by one; v) Repeat steps ⅰ)-ⅳ), converge all losses, and end the training until the preset maximum number of iterations is reached.

Images