Method and device for image deblurring based on generative adversarial network

Abstract

This invention provides a method and apparatus for image de-motion blurring based on generative adversarial networks (GANs). The method involves determining the application scenario based on processing requirements and acquiring a motion image within that scenario. The motion image is then input into a trained GAN to generate a first similar image with motion image features, which serves as the de-blurred motion image. The GAN in this invention uses a convolutional network for the generator's downsampling path. The original image is encoded through four convolutional downsampling passes, downsampling the spatial resolution along this path while extracting and compressing more semantic information. The upsampling path reconstructs a higher-resolution space from semantically rich layers, thereby capturing fine-grained details and coarse-grained semantics across all scales. Furthermore, three discriminators of different sizes perform multi-level discrimination of global and local details in the generator's generated image. Therefore, this invention achieves better image de-blurring results and has stronger scene adaptability.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method and apparatus for image de-motion blurring based on generative adversarial networks. Background Technology

[0002] Image motion blur removal aims to resolve image blur caused by camera or object movement, and has wide applications in security monitoring, medical imaging, and other fields. In video surveillance systems, movement of the camera or monitored object can lead to blurry images, potentially affecting security and the effectiveness of the monitoring system. Applying image motion blur removal technology can improve image quality, providing clearer visual information and aiding in target identification and tracking. In medical imaging, such as MRI or CT scans, motion caused by irregular breathing, heartbeat, or other movements of the patient can lead to image blur. Image motion blur removal can help doctors analyze images more accurately, resulting in more precise diagnoses. In photography, motion blur caused by camera shake or shooting fast-moving objects results in poor image quality. Image motion blur removal can improve image quality, enhance the clarity and usability of visual information, and play a crucial role in various application scenarios.

[0003] Existing methods for solving non-uniform motion fuzzy problems can be broadly divided into two types: one is an iterative optimization algorithm based on traditional degradation models, and the other is an algorithm that relies on neural network models.

[0004] Iterative optimization algorithms based on traditional degradation models typically model blurred images. Depending on whether the blur kernel is known, they can be divided into non-blind deblurring and blind deblurring. Non-blind deblurring algorithms assume the blur kernel is known; classic methods include Wiener filtering and the RL algorithm. The Wiener filtering algorithm adds external noise to the original data as input to the filter and deblurs by calculating the minimum mean square error between images. However, the restored image exhibits ghosting. The RL algorithm, based on Bayesian theory and assuming the blurred image follows a Poisson distribution, iteratively solves for the clear image using the maximum likelihood method. Its limitation lies in its susceptibility to local extrema, leading to ringing effects. LI Rudin et al. proposed an image deblurring algorithm based on a total variational model, further preserving image edge information. Levin et al. used maximum a posteriori estimation to repair images. Blind deblurring algorithms deblurr without knowing the blur kernel. The traditional algorithm's solution process generally consists of two stages: first, estimating the point spread function (PSF), i.e., the blur kernel; and second, estimating the clear image through deconvolution. Traditional methods often assume the blur is uniformly distributed across the entire image and then estimate the PSF. Fergus R et al. constructed a joint probability distribution of the sharp image and the blur kernel, and solved it iteratively. Cai et al. mapped the blur kernel to different domains and assumed that the blur kernel was sparse in different domains, thus using features in different domains as vividness adjustment to obtain a deblurring algorithm. Kim et al. used an image block method to estimate the spatially varying point spread function (PSF), and performed a more complex nonlinear PSF estimation by segmenting scene regions. Based on this, Kim and Lee used local linearity to approximate the PSF and simultaneously estimated the restored sharp image and the PSF representing local linear motion during the image restoration process. Jia et al. proposed that the edge information of the image is an important prior condition in motion-blurred images. Through manual annotation of edge information, they iterated and optimized continuously to finally obtain the image blur kernel. Pillai et al. used a multi-channel method to restore the image, generating more than two blurred images from a blurred image using an algorithm, and used a robust greatest common divisor algorithm to restore the degraded image from multiple generated blurred images. Xu et al. combined spatial prior conditions with Gaussian methods and used a cyclic iterative method to calculate the edge information of the image. Based on the Bayesian estimation algorithm, TSCho et al. extracted color level edge information of the image through the algorithm, then performed random transformation on the edge information, and reconstructed the blur kernel through a series of algorithms such as filtering and projection.

[0005] Xu et al. introduced sparse representation from signal processing to image deblurring. They mapped the image to a space composed of a set of orthogonal bases and obtained a sparse code for the image using an algorithm. The clear image was then reconstructed using this sparse code. Xiang et al. introduced dictionary learning into deblurring, using it to reduce the dimensionality of image data and learn image noise to achieve deblurring. Lin et al. used different directional filters for processing. They used multiple filter functions to estimate the blur kernel and obtain a clear image.

[0006] Methods for estimating blur kernels based on convolutional neural networks (CNNs) rely entirely on the dataset and do not consider the physical principles underlying the formation of the blur kernel. Sun et al. used CNNs to estimate the blur kernel and then deconvolved the blurred image with the predicted kernel. Chakrabarti et al. used CNNs to predict the Fourier coefficients of the blur kernel. Furthermore, Gong et al. used CNNs to estimate the motion flow of the blurred image to obtain a sharp image; this method is limited to linear blur kernels. However, the learning ability of neural networks allows them to not only replace one step in traditional deblurring but also handle image blurring caused by various factors end-to-end. In 2016, Nah et al. proposed using a multi-scale CNN to directly output the restored sharp image, replacing the traditional coarse-to-fine approach. In 2017, Noroozi et al. constructed an end-to-end deblurring network using a multi-scale pyramid structure combined with skip connections. Subsequently, Tao et al. applied recurrent neural networks to image restoration, proposing a scaled recurrent neural network combined with a pyramid structure to achieve image deblurring with a simpler network structure. Zhang et al. proposed a framework consisting of three deep convolutional networks and one recurrent neural network to perform feature map deconvolution operations, further improving the performance and speed of deep learning for image deblurring. Even so, these methods all relied on large datasets for training until Generative Adversarial Networks (GANs) such as DCGAN, LSGAN, and WGAN began to be applied to image processing. DCGAN replaced all fully connected layers with convolutional layers and all pooling layers with stride upsampling to increase stability, and used BatchNorm activation layers in each layer for feature normalization. However, the training results were highly unstable due to the difficulty in training GAN networks. Addressing the training difficulty of GAN networks, LSGAN used cross-entropy loss instead of least squares loss, and WGAN used Wasserstein distance to measure the difference between the generated data and the sample data to increase the stability of network training. In 2017, Isola et al. proposed the pix2pix model, applying conditional generative adversarial networks to image translation. The generator architecture used the U-Net architecture, and patchGAN was used as the discriminator. Kupyn et al. proposed DeblurGAN, whose network structure was based on conditional generative adversarial networks combined with adversarial and content loss, achieving good deblurring results. Zhang et al. proposed a deep CNN denoising model that provides priors for image reconstruction, using dilated convolutions instead of ordinary convolutions to expand the receptive field and using BN layers and residual block learning to accelerate training. Kupyn et al. further proposed DeblurGAN-v2 in 2019, which used a pyramid structure based on DeblurGAN.In 2022, Ali Syed Saqlain and others proposed Deblur-CycleGAN, introducing CycleGAN networks into the field of deblurring.

[0007] Traditional image deblurring algorithms often assume that the blur is uniformly distributed across the entire image, which limits their effectiveness. Non-blind deblurring algorithms can only generate blurred images within constrained scenarios to estimate the blur kernel, making them difficult to apply to real-world situations. Furthermore, image deblurring methods suffer from ghosting phenomena in image restoration, easily leading to local extrema and ringing effects. Blind deblurring algorithms suffer from inaccurate blur kernel estimation; the point spread function approximation method cannot completely estimate the true sharp blur kernel, especially in moving scenes with occlusion or sudden discontinuous motion, where the accuracy of the point spread function estimation directly affects the quality of the final sharp image. Edge information calibration is cumbersome, and iterative algorithms involve high computational costs.

[0008] Algorithms for image deblurring based on convolutional neural networks rely heavily on the dataset for estimating the blur kernel, without considering the underlying principles. They deconvolve the blurred image using the predicted kernel, leading to an over-reliance on the predicted kernel and requiring a large dataset for training. Algorithms based on generative adversarial networks (GANs) are prone to unstable outputs; even small changes in input can lead to different results, affecting the quality and consistency of the output image. Furthermore, the training results depend on the blur type and style of the dataset; overly simplistic datasets limit the application scenarios for deblurring. When the generator and discriminator are too complex and the dataset is insufficient, overfitting can easily occur, resulting in artifacts and inconsistencies in the final image.

[0009] In summary, existing deblurring solutions do not perform well in processing images, have long processing times, and are poorly adaptable to various motion blur types and scenes. Summary of the Invention

[0010] To address the aforementioned problems in the existing technology, this invention provides a method and apparatus for image de-motion blurring based on generative adversarial networks. The technical problem to be solved by this invention is achieved through the following technical solution:

[0011] In a first aspect, the present invention provides a method for image de-motion blurring based on generative adversarial networks, comprising:

[0012] S100: Determine the application scenario based on processing requirements and acquire motion images within that application scenario;

[0013] S200, the motion image is input into the trained generative adversarial network, so that the generative adversarial network uses its own generator to generate a first similar image with the features of the motion image;

[0014] The generator downsamples the spatial resolution of the motion image along a top-down downsampling path and extracts and compresses the semantic information of the motion image to obtain multiple first images of different sizes; it then performs multiple deconvolution upsampling and feature fusion on the first images of different sizes along a bottom-up upsampling path to generate the first similar image.

[0015] S300, the first similar image is determined as the motion image after blur removal.

[0016] Secondly, the present invention provides an apparatus for image de-motion blurring based on generative adversarial networks, comprising:

[0017] The acquisition module is configured to determine the application scenario based on processing requirements and acquire motion images within that application scenario;

[0018] The generation module is configured to input the motion image into a trained generative adversarial network, so that the generative adversarial network uses its own generator to generate a first similar image with the features of the motion image;

[0019] The generator downsamples the spatial resolution of the motion image along a top-down downsampling path and extracts and compresses the semantic information of the motion image to obtain multiple first images of different sizes; it then performs multiple deconvolution upsampling and feature fusion on the first images of different sizes along a bottom-up upsampling path to generate the first similar image.

[0020] The determination module is configured to determine the first similar image as a motion image after blur removal.

[0021] Beneficial effects:

[0022] This invention provides a method and apparatus for image de-motion blurring based on generative adversarial networks (GANs). The method involves determining the application scenario based on processing requirements and acquiring a motion image within that scenario. The motion image is then input into a trained GAN to generate a first similar image with motion image features, which serves as the de-blurred motion image. The GAN in this invention uses a convolutional network for the generator's downsampling path. The original image is encoded through four convolutional downsampling passes, downsampling the spatial resolution along this path while extracting and compressing more semantic information. The upsampling path reconstructs a higher-resolution space from semantically rich layers, thereby capturing fine-grained details and coarse-grained semantics across all scales. Furthermore, three discriminators of different sizes perform multi-level discrimination of global and local details in the generator's generated image. Therefore, this invention achieves better image de-blurring results and has stronger scene adaptability.

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a method for image de-motion blurring based on generative adversarial networks provided by the present invention.

[0025] Figure 2 This is a schematic diagram of the structure of the generative adversarial network provided by the present invention;

[0026] Figure 3 This is a feature map provided by the present invention. A diagram illustrating the construction process;

[0027] Figure 4 This is a schematic diagram of the training process provided by the present invention. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0029] like Figure 1 As shown, the present invention provides a method for image de-motion blurring based on generative adversarial networks, comprising:

[0030] S100: Determine the application scenario based on processing requirements and acquire motion images within that application scenario;

[0031] It's worth noting that processing requirements can be determined based on specific circumstances. If the processing requirement involves images for autonomous driving, the application scenario is defined as an autonomous driving environment. If the processing requirement involves images for community security, the application scenario is a security environment. If the processing requirement involves images for multi-object tracking, the application scenario is a multi-object tracking environment.

[0032] S200, the motion image is input into the trained generative adversarial network, so that the generative adversarial network uses its own generator to generate a first similar image with the features of the motion image;

[0033] The generator downsamples the spatial resolution of the motion image along a top-down downsampling path and extracts and compresses the semantic information of the motion image to obtain multiple first images of different sizes; it then performs multiple deconvolution upsampling and feature fusion on the first images of different sizes along a bottom-up upsampling path to generate the first similar image.

[0034] S300, the first similar image is determined as the motion image after blur removal.

[0035] This invention provides a method for image de-motion blurring based on generative adversarial networks (GANs). The method determines the application scenario based on processing requirements and acquires a motion image within that scenario. The motion image is then input into a trained GAN to generate a first similar image with motion image features, which serves as the de-blurred motion image. The GAN in this invention uses a convolutional network for the generator's downsampling path. The original image is encoded through four convolutional downsampling passes, downsampling the spatial resolution along this path while extracting and compressing more semantic information. The upsampling path reconstructs a higher-resolution space from semantically rich layers, thereby capturing fine-grained details and coarse-grained semantics across all scales. Furthermore, three discriminators of different sizes perform multi-level discrimination of global and local details in the generator's generated image. Therefore, this invention achieves better image de-blurring results and has stronger scene adaptability.

[0036] refer to Figure 2 The generative adversarial network of this invention includes a generator and a discriminator. The generator employs an optimized encoding / decoding structure, which introduces full-size skip connections based on a feature pyramid structure. The discriminator employs a multi-discrimination mechanism, internally including multiple discriminators. The discriminator's evaluation function is opposite to that of the generator, forming a minimax game.

[0037] This invention employs a generative adversarial network (GAN), whose architecture consists of a generator and a discriminator. The network is trained using paired datasets. The generator learns the features of the training set data and, guided by the discriminator, tries to fit the random noise distribution as closely as possible to the true distribution of the training data, thus generating similar images with the features of the training set. The discriminator aims to distinguish between truly clear images and images generated by the generator, determining whether the input is real or fake data generated by the generator. The two networks are trained alternately until the data generated by the generator can be indistinguishable from real data. The training objective of the discriminator D is to maximize the classification accuracy, while the training objective of the generator G is to minimize the accuracy of the discriminator D. Their evaluation functions are completely opposite, thus forming a minimax game process, represented as:

[0038]

[0039] Where D(x) represents the probability that the discriminator classifies x as real data, G(z) represents the output of the generator, and D(G(z)) represents the probability that the discriminator classifies G(z) as real data.

[0040] The generator employs an encoding / decoding structure, introducing full-size skip connections on top of the feature pyramid structure. The discriminator uses a multi-discrimination mechanism, which helps guide the generator to further recover global and local detail information. An edge loss function is introduced to further recover edge information, and L2 regularization is used to prevent image overfitting. The dataset utilizes rich application scenarios and diverse motion blur forms to generate pairs of blurred and sharp images for training. This achieves a sharp deblurring effect on any input motion-blurred image.

[0041] In one specific embodiment of the present invention, the generator includes four encoders and four decoders. The four encoders are connected sequentially from top to bottom to form an upsampling path, and the decoders are connected sequentially from bottom to top to form a downsampling path. The encoders and decoders in the same layer are connected, and the motion image or sample pair input to the generative adversarial network is the original input image.

[0042] The first layer encoder processes the input image. Spatial resolution and extraction and compression of the input original image The semantic information is used to obtain the feature map. Each layer's encoder takes the spatial resolution of the feature map output by the previous layer's encoder, extracts and compresses the semantic information of the feature map to obtain the feature map of this layer, and outputs it to the next layer; the feature maps output by the first to fourth encoders are respectively in, To input 1 / 4 of the original image, To input 1 / 8 of the original image, To input 1 / 16 of the original image, Input 1 / 32 of the original image;

[0043] The fourth layer decoder pair The feature map is obtained by deconvolution upsampling the feature map. The feature map is passed to the third-layer decoder; the third-layer decoder receives the feature map. And on The feature information obtained from the four methods, including pooling and downsampling to unify the feature map resolution, is then fused to obtain the final feature map. Transmitted to the second-layer decoder;

[0044] The second-layer decoder receives the feature map. And on The feature information obtained from the three methods (pooling and downsampling to unify the feature map resolution) is then fused to obtain the final feature map. The feature map is passed to the first-layer decoder; the first-layer decoder receives the feature map. And on Pooling and downsampling are performed to unify the feature map resolution, and the feature information from the three sources is then fused and deconvolution upsampling is performed to obtain similar images.

[0045] This invention employs a full-size skip-level connection generator with an optimized feature pyramid structure, including both downsampling and upsampling paths. The specific structure is as follows: Figure 2 As shown, the downsampling path uses a convolutional network. The original image is encoded by four convolutional downsampling passes. The spatial resolution is downsampled along this path, while extracting and compressing more semantic information, resulting in five feature maps of different sizes. in, To input the original image, It is 1 / 4 of the original image. It is 1 / 8 of the original image. It is 1 / 16 of the original image. It is 1 / 32 of the original image. The upsampling path reconstructs a higher resolution space from semantically rich layers, including the lowest layer. The feature map is obtained by deconvolution upsampling the feature map. By directly receiving feature maps from encoder layers of the same scale The feature information, and will After pooling downsampling to unify the feature map resolution, the feature information obtained is fused to obtain the final feature map. Then, deconvolution upsampling is performed to obtain the final feature map. Similarly Receive feature information from feature map E3 of the encoder layer at the same scale, and simultaneously... Pooling downsampling is performed to unify the feature map resolution before feature fusion is performed to obtain the final feature map. Then, deconvolution upsampling is performed to obtain the final feature map. Encoder layers of the same scale After pooling downsampling Feature fusion is performed, followed by deconvolution and upsampling to obtain the final generator output image. Each decoder layer combines small-scale and same-scale feature maps from the encoder with large-scale feature maps from the decoder, thereby capturing fine-grained details and coarse-grained semantics across the entire scale.

[0046] The following formula represents a full-size skip connection, where i represents the i-th downsampling layer along the encoding direction, N represents the number of encoders, and the feature map... The calculation formula is as follows:

[0047]

[0048] Where i represents the i-th layer along the encoding direction, N represents the number of encoders, and C represents convolution. Indicates downsampling, Indicates upsampling, This represents a feature aggregation mechanism, which includes a convolutional layer, a batch normalization (BN) layer, and a ReLU function. This indicates the encoder.

[0049] The network structure used in this invention is more efficient with fewer parameters. In the encoder subnet, E i The number of channels is 32×2 i The number of parameters in the i-th decoder stage can be expressed by the formula:

[0050]

[0051] Among them, D F The convolution kernel size is represented by d(·), and the node depth is represented by d(·). Indicates decoder, This indicates the encoder.

[0052] refer to Figure 3 , Figure 3 It explains in detail how to build Feature maps from encoder layers of the same size. The feature maps are received directly in the decoder, and a set of skip connections between encoders and decoders are achieved by applying non-overlapping max-pooling operations, starting from a smaller encoder layer. Provides low-level details, while intra-decoder hop connections utilize bilinear interpolation from larger-scale decoder layers. and To transmit high-level semantic information, 64 filters of size 3*3 are used for convolution to further unify the number of channels. At the same time, a feature aggregation mechanism is performed on the connected feature maps at 5 scales. The feature maps are then passed through a convolution + BN + ReLU block to output features.

[0053] In one specific embodiment of the present invention, the discriminator employs three discriminators D1, D2, and D3 at different scales to downsample the clear image and the similar image generated by the generator in the sample pair by 1 / 2 and 1 / 4, forming an image pyramid at three scales. The discriminator operating at the coarsest scale has the largest perceptual range, while the discriminator operating at the finest scale guides the generator to produce finer image details.

[0054] The discriminator employs three models at different scales (D1, D2, and D3), downsampling the real, sharp image and the deblurred image generated by the generator by 1 / 2 and 1 / 4, respectively, to form an image pyramid at three scales. Then, discriminators D1, D2, and D3 are trained separately to distinguish between real and synthetic images at three different scales. Although the three discriminators have similar structures, the discriminator operating at the coarsest scale has the largest perceptual range, thus acquiring a more global image perspective and guiding the generator to produce globally consistent images. Meanwhile, the discriminator operating at the finest scale focuses on guiding the generator to produce finer details. This design not only makes training the generator from coarse to finer scales easier but also simplifies scaling low-resolution models to higher resolutions, requiring only the addition of an additional discriminator at the finest level without retraining the entire model from scratch. Based on formula (1), the game between the discriminator and the generator with the introduction of three discriminators can be expressed by the formula:

[0055]

[0056] The goal of GAN is to find the parameters of the generator G and the discriminator D such that the loss of the discriminator is maximized while the loss of the generator is minimized.

[0057] This invention introduces adversarial loss, edge loss, and L2 regularization loss to measure the generation loss of the generator.

[0058] Adversarial loss is represented as:

[0059]

[0060] Where D(x) represents the probability that the discriminator classifies x as real data, G(z) represents the output of the generator, and D(G(z)) represents the probability that the discriminator classifies G(z) as real data.

[0061] Since edge information in an image is generally high-frequency information, but the high-frequency components change during image blurring, edge loss is introduced to further improve the recovery of structural and texture details. The edge loss function can be expressed as:

[0062]

[0063] Among them, I real I represents the edge information of a real image. fake This represents the edge information of the image generated by the generator.

[0064] During network training, the weights in certain features of the training dataset may become extremely large, leading to overfitting. Huge weights are highly susceptible to significant output fluctuations with small input changes in neural networks. Even two very similar independent data inputs can result in large differences in their outputs, causing numerous incorrect predictions on test data and reducing the network's generalization ability. This study introduces L2 regularization as the generator's loss function to reduce model complexity and prevent overfitting. The L2 regularization loss function is expressed as follows:

[0065]

[0066] Where x i This represents the i-th real image. This represents the image generated by the i-th generator.

[0067] Therefore, the overall expression for the generator's loss function can be represented as:

[0068]

[0069] in, For counter-loss, For edge loss, λ1, λ2, and λ3 are L2 regularization loss parameters.

[0070] In one specific embodiment of the present invention, the generative adversarial network is trained using a training dataset, and the process of generating the training dataset includes:

[0071] a1 uses the GOPRO dataset in conjunction with the Real Blur Dataset dataset as the dataset; all images in the dataset are clear images.

[0072] This step uses the GOPRO dataset in conjunction with the Real Blur Dataset as the sharp images for this study, with a uniform image size of 1280*720. By adding different forms of motion blur to the images, corresponding blurred images are created, thus generating a training dataset with a one-to-one correspondence between sharp and blurred images. The final dataset includes 8000 image pairs across 265 application scenarios. The training set consists of 6500 image pairs, and the test set consists of 1500 image pairs.

[0073] a2, unify the size of all images in the dataset to 1280*720;

[0074] a3, add different forms of motion blur to images of the same size to create corresponding blurred images;

[0075] a4 sets up sample pairs consisting of all clear images and their corresponding blurry images, and all sample pairs together form the training dataset.

[0076] In one specific embodiment of the present invention, reference is made to... Figure 4 The process of training the generative adversarial network includes:

[0077] b1. For any sample pair in the training set, the sample pair is input into a preset generative adversarial network (GAN), so that the GAN uses its generator to downsample the spatial resolution of the sample pair along a top-down downsampling path, and extracts and compresses the semantic information of the sample pair to obtain multiple second images of different sizes; and performs multiple deconvolution upsampling and feature fusion on the second images of different sizes along a bottom-up upsampling path to generate a second similar image with clear image features from the sample pair.

[0078] Continue to refer to Figure 2The generator uses an encoder-decoder structure. The encoder consists of convolutional layers, batch normalization, ReLU layers, and pooling layers, while the decoder consists of convolutional layers, batch normalization, ReLU layers, and upsampling. Taking decoder 3 as an example, decoder 3 fuses features from encoders 1, 2, and 3, and decoders 4 and 5. These features are adjusted to the same feature map size as decoder 3 through max pooling (features from the encoder) or upsampling (features from the decoder), and the number of features is adjusted to be consistent through convolutional layers (convolution + BN + ReLU). These concatenated feature maps are then processed by a convolution + BN + ReLU block to output features. The discriminator consists of three downsampling convolutional layers at three scales. Using gradient descent, the weight parameters of the adversarial loss function, marginal loss, and L2 regularization loss function are determined to obtain the joint loss function. The loss of the discriminator network and the loss generated in the generator network are trained alternately to obtain the adversarial network model.

[0079] b2, the second similar image is used as the deblurring result of the blurred image in the sample pair;

[0080] b3, The generative adversarial network uses its own discriminator to distinguish the second similar image and uses the discrimination result to guide the generator to adjust its own parameters in order to generate a globally consistent third similar image;

[0081] The discriminator downsamples the second similar image and the corresponding clear image to obtain synthetic images of different sizes; and uses three discriminators of different scales to distinguish the synthetic images to obtain the discrimination results; based on the discrimination results and the generator's loss function, the generator is guided to adjust its own parameters to generate the globally consistent third similar image.

[0082] b4, repeat b1 to b3 until all samples are used as input to obtain the trained adversarial generative network.

[0083] This step fixes the network weights and uses the established training set of 6500 pairs to train the generative adversarial network model. The established test set of 1500 pairs is then used to validate the model on the trained network, yielding the deblurring results. Blurred images of real-world scenes are captured and validated on the network. The results show that, compared to the DeblurGAN-V2 network algorithm, the algorithm used in this invention restores clearer image edge contours, better recovers image detail information, and has better generalization ability.

[0084] This invention uses end-to-end mapping, eliminating the need for iterative estimation of the blur kernel, thus overcoming the drawbacks of traditional algorithms that require iterative estimation of the blur kernel and shortening the deblurring time. It introduces full-size skip connections to extract and compress more semantic information, capturing more fine-grained details and coarse-grained semantic information across all scales. Using a multi-scale discriminator and introducing an edge loss function reduces the ringing effect during image restoration and also reduces the generation of artifacts, improving the clarity of the restored image. Furthermore, it uses a dataset with richer scene representations for training and introduces the L2 norm to prevent overfitting while further improving the network's generalization ability.

[0085] This invention uses generative adversarial networks (GANs) for image deblurring. Unlike previous GAN deblurring methods, the generator employs full-size skip-level connections, obtaining feature maps of five sizes through downsampling and upsampling. The downsampling path uses a convolutional network, encoding the original image through four convolutional downsampling passes. This path downsamples the spatial resolution while simultaneously extracting and compressing more semantic information, resulting in five feature maps of different sizes. The upsampling path reconstructs a higher-resolution space from semantically rich layers. Each decoder layer combines small-scale and same-scale feature maps from the encoder with large-scale feature maps from the decoder, thereby capturing fine-grained details and coarse-grained semantics across the entire scale. A multi-scale discriminator with three sizes enables multi-discrimination of global and local details. An edge loss function and an L2 regularization loss function are introduced to further improve edge information recovery and enhance the network's generalization ability. Therefore, this invention achieves superior image deblurring results.

[0086] Secondly, the present invention provides an apparatus for image de-motion blurring based on generative adversarial networks, comprising:

[0087] The acquisition module is configured to determine the application scenario based on processing requirements and acquire motion images within that application scenario;

[0088] The generation module is configured to input the motion image into a trained generative adversarial network, so that the generative adversarial network uses its own generator to generate a first similar image with the features of the motion image;

[0089] The generator downsamples the spatial resolution of the motion image along a top-down downsampling path and extracts and compresses the semantic information of the motion image to obtain multiple first images of different sizes; it then performs multiple deconvolution upsampling and feature fusion on the first images of different sizes along a bottom-up upsampling path to generate the first similar image.

[0090] The determination module is configured to determine the first similar image as a motion image after blur removal.

[0091] This invention provides an apparatus for image de-motion blurring based on a generative adversarial network (GAN). The application scenario is determined according to processing requirements, and a motion image within that scenario is acquired. This motion image is then input into a trained GAN to generate a first similar image with motion image features, which serves as the de-blurred motion image. The GAN in this invention uses a convolutional network for the generator's downsampling path. The original image is encoded through four convolutional downsampling passes, downsampling the spatial resolution along this path while extracting and compressing more semantic information. The upsampling path reconstructs a higher-resolution space from semantically rich layers, thereby capturing fine-grained details and coarse-grained semantics across all scales. Furthermore, three discriminators of different sizes perform multi-discrimination of global and local details in the generator's generated image. Therefore, this invention provides better image de-blurring performance and stronger scene adaptability.

[0092] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0093] Although this application has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the appended claims in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality.

[0094] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for image de-motion blurring based on generative adversarial networks, characterized in that, include: S100: Determine the application scenario based on processing requirements and acquire motion images within that application scenario; S200, the motion image is input into the trained generative adversarial network, so that the generative adversarial network uses its own generator to generate a first similar image with the features of the motion image; The generator downsamples the spatial resolution of the motion image along a top-down downsampling path and extracts and compresses the semantic information of the motion image to obtain multiple first images of different sizes; it then performs multiple deconvolution upsampling and feature fusion on the first images of different sizes along a bottom-up upsampling path to generate the first similar image. S300, the first similar image is determined as the motion image after blur removal; The generator includes four encoders and four decoders. The four encoders are connected sequentially from top to bottom to form an upsampling path, and the decoders are connected sequentially from bottom to top to form a downsampling path. The encoders and decoders in the same layer are connected, and the motion image or sample pair input to the generative adversarial network is the original input image. The first layer encoder processes the input image. Spatial resolution and extraction and compression of the input original image The semantic information is used to obtain the feature map. Each layer's encoder takes the spatial resolution of the feature map output by the previous layer's encoder, extracts and compresses the semantic information of the feature map to obtain the feature map of the current layer, and outputs it to the next layer; the feature maps output by the first to fourth encoders are respectively , , , ,in, To input 1 / 4 of the original image, To input 1 / 8 of the original image, To input 1 / 16 of the original image, Input 1 / 32 of the original image; The fourth layer decoder pair The feature map is obtained by deconvolution upsampling the feature map. The feature map is passed to the third-layer decoder; the third-layer decoder receives the feature map. and to , , , The feature information obtained from the four methods, including pooling and downsampling to unify the feature map resolution, is then fused to obtain the final feature map. Transmitted to the second-layer decoder; The second-layer decoder receives the feature map. and to , , The feature information obtained from the three methods (pooling and downsampling to unify the feature map resolution) is then fused to obtain the final feature map. The feature map is passed to the first-layer decoder; the first-layer decoder receives the feature map. and to , Pooling and downsampling are performed to unify the feature map resolution, and the feature information from the three sources is then fused and deconvolution upsampling is performed to obtain similar images. ; The feature map The calculation formula is expressed as: in, i Indicates the first [number] along the encoding direction i layer, N Indicates the number of encoders. Represents convolution. Indicates downsampling, Indicates upsampling, This represents a feature aggregation mechanism, which includes a convolutional layer, a batch normalization (BN) layer, and a ReLU function. Indicates encoder; In the encoder subnet, the i-th encoder The number of channels is The number of parameters in the i-th decoder stage can be expressed by the formula: in, Indicates the size of the convolution kernel. Indicates the depth of the node. Indicates decoder, This indicates the encoder.

2. The method for image de-motion blurring based on generative adversarial networks according to claim 1, characterized in that, The generative adversarial network includes a generator and a discriminator. The generator adopts an optimized encoding and decoding structure, which introduces full-size skip connections on the basis of the feature pyramid structure. The discriminator adopts a multi-discrimination mechanism and includes multiple discriminators internally.

3. The method for image de-motion blurring based on generative adversarial networks according to claim 1, characterized in that, The generative adversarial network is trained using a training dataset, and the process of generating the training dataset includes: a1 uses the GOPRO dataset in conjunction with the Real Blur Dataset dataset as the dataset; all images in the dataset are clear images; a2, unify the size of all images in the dataset to 1280*720; a3, add different forms of motion blur to images of the same size to create corresponding blurred images; a4 sets up sample pairs consisting of all clear images and their corresponding blurry images, and all sample pairs together form the training dataset.

4. The method for image de-motion blurring based on generative adversarial networks according to claim 3, characterized in that, The process of training the generative adversarial network includes: b1. For any sample pair in the training dataset, the sample pair is input into a preset generative adversarial network (GAN), so that the GAN uses its generator to downsample the spatial resolution of the sample pair along a top-down downsampling path, and extracts and compresses the semantic information of the sample pair to obtain multiple second images of different sizes; multiple deconvolution upsampling and feature fusion are performed on the second images of different sizes along a bottom-up upsampling path to generate a second similar image with clear image features from the sample pair. b2, the second similar image is used as the deblurring result of the blurred image in the sample pair; b3, The generative adversarial network uses its own discriminator to distinguish the second similar image and uses the discrimination result to guide the generator to adjust its own parameters in order to generate a globally consistent third similar image; The discriminator downsamples the second similar image and the corresponding clear image to obtain synthetic images of different sizes; and uses three discriminators of different scales to distinguish the synthetic images to obtain the discrimination results; based on the discrimination results and the generator's loss function, the generator is guided to adjust its own parameters to generate the globally consistent third similar image. b4, repeat b1 to b3 until all samples are used as input to obtain the trained adversarial generative network.

5. The method for image de-motion blurring based on generative adversarial networks according to claim 4, characterized in that, The discriminator and the generator have opposite evaluation functions, forming a minimax game, as shown below: ； in, This represents the probability that the discriminator classifies x as real data. This represents the output of the generator. Indicates the discriminator's discrimination. The probability of the actual data.

6. The method for image de-motion blurring based on generative adversarial networks according to claim 4, characterized in that, The loss function of the generator is expressed as follows: ； in, For counter-loss, For edge loss, For L2 regularization loss, , , These are the loss weight parameters.

7. The method for image de-motion blurring based on generative adversarial networks according to claim 4, characterized in that, The discriminator employs three discriminators D1, D2, and D3 at different scales. It downsamples the clear image and the similar image generated by the generator in the sample pair by 1 / 2 and 1 / 4, forming an image pyramid at three scales. The discriminator operating at the coarsest scale has the largest perceptual range, while the discriminator operating at the finest scale guides the generator to produce finer image details.

8. An apparatus for image de-motion blurring based on generative adversarial networks, characterized in that, include: The acquisition module is configured to determine the application scenario based on processing requirements and acquire motion images within that application scenario; The generation module is configured to input the motion image into a trained generative adversarial network, so that the generative adversarial network uses its own generator to generate a first similar image with the features of the motion image; The generator downsamples the spatial resolution of the motion image along a top-down downsampling path and extracts and compresses the semantic information of the motion image to obtain multiple first images of different sizes; it then performs multiple deconvolution upsampling and feature fusion on the first images of different sizes along a bottom-up upsampling path to generate the first similar image. The determining module is configured to determine the first similar image as a motion image after blur removal; The generator includes four encoders and four decoders. The four encoders are connected sequentially from top to bottom to form an upsampling path, and the decoders are connected sequentially from bottom to top to form a downsampling path. The encoders and decoders in the same layer are connected, and the motion image or sample pair input to the generative adversarial network is the original input image. The first layer encoder processes the input image. Spatial resolution and extraction and compression of the input original image The semantic information is used to obtain the feature map. Each layer's encoder takes the spatial resolution of the feature map output by the previous layer's encoder, extracts and compresses the semantic information of the feature map to obtain the feature map of the current layer, and outputs it to the next layer; the feature maps output by the first to fourth encoders are respectively , , , ,in, To input 1 / 4 of the original image, To input 1 / 8 of the original image, To input 1 / 16 of the original image, Input 1 / 32 of the original image; The fourth layer decoder pair The feature map is obtained by deconvolution upsampling the feature map. The feature map is passed to the third-layer decoder; the third-layer decoder receives the feature map. and to , , , The feature information obtained from the four methods, including pooling and downsampling to unify the feature map resolution, is then fused to obtain the final feature map. Transmitted to the second-layer decoder; The second-layer decoder receives the feature map. and to , , The feature information obtained from the three methods (pooling and downsampling to unify the feature map resolution) is then fused to obtain the final feature map. The feature map is passed to the first-layer decoder; the first-layer decoder receives the feature map. and to , Pooling and downsampling are performed to unify the feature map resolution, and the feature information from the three sources is then fused and deconvolution upsampling is performed to obtain similar images. ; The feature map The calculation formula is expressed as: in, i Indicates the first [number] along the encoding direction i layer, N Indicates the number of encoders. Represents convolution. Indicates downsampling, Indicates upsampling, This represents a feature aggregation mechanism, which includes a convolutional layer, a batch normalization (BN) layer, and a ReLU function. Indicates encoder; In the encoder subnet, the i-th encoder The number of channels is The number of parameters in the i-th decoder stage can be expressed by the formula: in, Indicates the size of the convolution kernel. Indicates the depth of the node. Indicates decoder, This indicates the encoder.

Images