Methods, devices, and media for removing motion blur from images based on GANs

Abstract

This invention discloses a method, apparatus, and medium for removing motion blur from images based on a Generative Adversarial Network (GAN). The method includes: acquiring a training set consisting of multiple blurred images and corresponding clear images, wherein the multiple blurred images include images with uneven motion blur; training the generator and discriminator of a GAN model based on the training set, such that the features obtained after passing the blurred images through the generator and discriminator are consistent with the features obtained after passing the corresponding clear images through the discriminator; extracting the trained generator from the trained GAN model, and using the trained generator to deblur the image to be deblurred. This embodiment removes uneven motion blur from images.

Description

Technical Field

[0001] The embodiments of the present invention relate to fields such as artificial intelligence and security monitoring, and in particular to a method, device and medium for removing motion blur from images based on GAN. Background Technology

[0002] Image deblurring is a classic problem in computer vision, aiming to recover a sharp image from a blurred input image. There are many types of image blurring, with motion blur being the most common. Causes of motion blur in traffic include camera shake, excessive vehicle speed, and defocusing.

[0003] In existing technologies, excessive vehicle speed can be addressed by increasing the shutter speed for image capture, but this inevitably increases the hardware cost of the entire highway network. Another type of deblurring method based on deep learning typically estimates a blur kernel first and then uses the kernel for inverse operations to remove blur. This method has the following drawbacks: firstly, the blur kernel is difficult to obtain; secondly, the blur kernel generally only removes uniform blur. In the real world, such as at intersections, pedestrians and vehicles move in all directions, and combined with camera shake, this results in unevenly blurred images. Traditional methods for estimating blur kernels cannot effectively solve this problem. Summary of the Invention

[0004] This invention provides a method, device, and medium for removing motion blur from images based on GAN, which effectively removes uneven motion blur in images.

[0005] In a first aspect, embodiments of the present invention provide a method for removing motion blur from images based on GAN, comprising:

[0006] Obtain a training set consisting of multiple blurred images and corresponding clear images, wherein the multiple blurred images include non-uniform motion blurred images;

[0007] The generator and discriminator of the GAN model are trained based on the training set, so that the features obtained by the generator and discriminator after the blurred image passes through the blurry image tend to be consistent with the features obtained by the discriminator after the corresponding clear image passes through the blurry image.

[0008] Extract the trained generator from the trained GAN model, and use the trained generator to deblur the image that needs to be deblurred.

[0009] The generator adopts a U-net structure, which includes a symmetrical encoder and decoder. The encoder includes alternating multi-level downsampling operations and StageBlocks, and the decoder includes alternating multi-level upsampling operations and StageBlocks. Each StageBlock in the encoder is residually connected to each StageBlock in the decoder.

[0010] Each StageBlock includes a parallel Transformer module and a frequency domain processing module. The Transformer module is used to extract the spatial domain features of the image, and the frequency domain processing module is used to filter the frequency domain information of the image. The outputs of the Transformer module and the frequency domain processing module are added together to serve as the output of each StageBlock.

[0011] In a second aspect, embodiments of the present invention provide an electronic device, comprising:

[0012] One or more processors;

[0013] Memory, used to store one or more programs.

[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-described method for removing motion blur from images based on GANs.

[0015] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for removing motion blur from images based on GAN.

[0016] This invention proposes a method for removing motion blur from images using an end-to-end GAN model. This method restores images blurred by factors such as object motion, camera shake, defocusing, and atmospheric turbulence, resulting in clear and recognizable images. This method eliminates the need to estimate the blur kernel, reducing computational complexity, and can handle uneven motion blur, broadening its application scope. Specifically, this method combines a GAN model with a Transformer. The GAN model only uses backpropagation, eliminating the need for complex Markov chains, and can generate the most realistic images among various generative models, effectively restoring image texture details. The Transformer, on the other hand, has powerful modeling capabilities, and its performance improves with larger datasets. Increased datasets themselves enhance the accuracy of model simulation, thus significantly improving the deblurring effect. Furthermore, in this embodiment, StageBlock extracts spatial features using the Transformer module, filters frequency domain information using a frequency domain processing module, and combines the outputs of the Transformer module and the frequency domain processing module for image deblurring. Since deblurring in the spatial domain can easily lose image detail information, while the frequency domain can preserve image texture details, but the image edges in the frequency domain are prone to oversmoothing, this embodiment combines the spatial and frequency domains to deblur the image, which can balance the advantages and disadvantages of spatial and frequency domain processing, compensate for each other's shortcomings, and improve the final deblurring effect. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a method for removing motion blur from images based on GAN, provided in an embodiment of the present invention.

[0019] Figure 2 This is a flowchart of another method for removing motion blur from images based on GAN, provided in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the structure of a generator provided in an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of a StageBlock provided in an embodiment of the present invention.

[0022] Figure 5This is a schematic diagram of the structure of a Block in a Transformer module provided in an embodiment of the present invention.

[0023] Figure 6 This is a comparison diagram in the spatial and frequency domains of the blurred image and the deblurred image provided in the embodiments of the present invention.

[0024] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0026] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0027] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0028] Figure 1 This is a flowchart of a GAN-based method for removing motion blur from images, provided by an embodiment of the present invention. This method is primarily used to address the blurring problem of fast-moving objects in images, aiming to remove the ghosting caused by fast-moving objects and obtain a clear image. This method can remove image blur not only caused by uniform motion but also by non-uniform motion. This method is executed by an electronic device, such as... Figure 1 As shown, the specific steps include the following.

[0029] S110. Obtain a training set consisting of multiple blurred images and corresponding clear images, wherein the multiple blurred images include non-uniform motion blurred images.

[0030] In subsequent steps, this embodiment will utilize a deep learning network model to restore the blur caused by the high-speed movement of objects when capturing images, and the training set will serve as the sample set for training the network model.

[0031] Optionally, the training set can be constructed as follows: multiple original blurred images and a corresponding sharp image for each original blurred image are obtained; each original blurred image and sharp image are cut and recombined to form the training set. Both the original blurred images and their corresponding sharp images are large-sized images. In this embodiment, large-sized images are cut into small pieces, and the training set is formed from these cut image pieces, which can reduce the burden on the model and lower hardware training costs.

[0032] Specifically, both the original blurred image and its corresponding sharp image can be divided into 256×256 blocks. To increase the dataset size, three basic operators—translation, cropping, and color modification—are used to augment the segmented image blocks. Translation moves the image blocks along the x-axis or y-axis (or two other directions); during translation, the background is assumed to be black because a portion of the image is empty during translation. Cropping is not only a data augmentation method but also a way to reduce data noise and increase model stability, thus improving model accuracy. Random cropping establishes a weight relationship between each factor feature and its corresponding category, weakening the weight of background (or noise) factors and making the model less sensitive to missing values, resulting in better learning performance and increased model stability. Color modification modifies image blocks by changing hue, saturation, brightness, etc., to serve as new training data, allowing the model to better process images under different lighting conditions.

[0033] S120. The generator and discriminator of the GAN model are trained based on the training set, so that the features obtained by the generator and discriminator after the blurred image passes through the blurred image tend to be consistent with the features obtained by the discriminator after the corresponding clear image passes through the clear image.

[0034] The image deblurring method in this embodiment is implemented based on the GAN (Generative Adversarial Network) model. Figure 2 This is a flowchart of another method for removing motion blur from images based on GAN, provided in an embodiment of the present invention. Figure 2As shown, the GAN model includes a generator and a discriminator. The generator is used to restore the blurred image to obtain a deblurred image. The discriminator is used to calculate the W distance between the deblurred image and the clear image, thereby calculating the loss function. By minimizing the loss function, the features obtained from the blurred image after passing through the generator and discriminator are made to be consistent with the features obtained from the corresponding clear image after passing through the discriminator, thus completing the training of the generator and discriminator.

[0035] Furthermore, Figure 3 This is a schematic diagram of the structure of a generator provided in an embodiment of the present invention. Figure 3 As shown, the generator adopts a U-net structure, which includes a symmetrical decoder and encoder, with a StageBlock positioned between the decoder and encoder. The output of the decoder serves as the input to the StageBlock, and the output of the StageBlock serves as the input to the encoder. More specifically, the encoder includes alternating multi-level downsampling operations and StageBlocks. The output of the current downsampling operation is the input to the current StageBlock, the output of the current StageBlock is the input to the next downsampling operation, and so on. Similarly, the decoder includes alternating multi-level upsampling operations and StageBlocks. The output of the current upsampling operation is the input to the current StageBlock, the output of the current StageBlock is the input to the next upsampling operation, and so on. Furthermore, each StageBlock in the encoder is residually connected to each StageBlock in the decoder. Optionally, as... Figure 3 As shown, assuming that the decoder and encoder each include M StageBlocks, the output of the i-th StageBlock in the decoder is added to the input of the M-th StageBlock in the encoder, and then used as the input of the (i+1)-th StageBlock in the decoder, where M is a natural number greater than 1, and i takes the values ​​1, 2, ..., M in sequence.

[0036] StageBlock is used for feature extraction from images. Figure 4 This is a schematic diagram of a StageBlock structure provided in an embodiment of the present invention. Figure 4As shown, each StageBlock includes a parallel Transformer module and a frequency domain processing module. The Transformer module is used to extract the spatial domain features of the image, and the frequency domain processing module is used to filter the frequency domain information of the image. The outputs of the Transformer module and the frequency domain processing module are added together to serve as the output of each StageBlock.

[0037] based on Figures 2 to 4 The training process of the GAN model shown includes the following steps:

[0038] Step 1: Input any blurred image from the training set into the generator of the GAN model for deblurring. Optionally, first, input the blurred image into the encoder, where multi-level downsampling and feature extraction are performed alternately. Taking a 256×256 input image as an example, since the GAN model uses the Transformer attention mechanism, a 256×256 image, even after flattening, is still an extremely long sequence of 65536. Therefore, the Unet structure is used to downsample it first before performing the complete attention operation, which keeps the model's time complexity within an acceptable range. Furthermore, since Transformer models are highly dependent on data volume, as the data volume increases, Transformer gradually catches up with or even surpasses CNN. This is the reason for data augmentation in S110; augmentation significantly increases the data volume, effectively utilizing the performance advantage of Transformer under large data volumes.

[0039] In one specific implementation, a convolution operation is performed on any blurred image in the training set to reduce the image size and expand the channel dimension. Taking a 256×256 blurred image as an example (e.g....),... Figure 3 As shown), the current image is represented as I. R If the batch size is set to 1, then the dimensions of the current image are 1×256×256×3. Figure 3 (The batch size dimension is omitted). Convolution operations are used to reduce the image size (I). R The size is increased and the channels are expanded to obtain image I1:

[0040] I1=I R *θ1 (1)

[0041] Where * denotes the convolution operation, θ1 represents the convolution kernel, and I1 represents the image after reducing its size and expanding its channels. It's worth noting that after the convolution operation, the number of channels in the image increases from I... R The 3 channels of I1 are changed to 16 channels of I1.

[0042] After the convolution operation, the resulting image is input into the encoder as the current image; operations S1-1, S1-2, and S1-3 are performed in the encoder:

[0043] S1-1. Perform the inverse operation of Pixel Shuffle on the current image to further reduce the image size and expand the channel dimensions. The inverse operation of Pixel Shuffle constitutes the current level of downsampling operation, and its output is the output of the current level of downsampling operation. Taking I1 as an example of the image after convolution, the inverse operation of Pixel Shuffle halves the size of image I1 and quadruples the number of channels, resulting in a result I2 with dimensions of 1×128×128×64. The main function of Pixel Shuffle is to obtain a high-resolution feature map from a low-resolution feature map through convolution and multi-channel recombination. It can convert H×W×C×r... 2 The tensor transformation is rH×rW×C, and its formula is expressed as:

[0044]

[0045] This completes the current level downsampling operation, and I2 is the output of the current level downsampling operation.

[0046] S1-2. The output of the current level downsampling operation is input into the Transformer module and the frequency domain processing module respectively, to extract spatial domain features and filter frequency domain information. The Transformer module and the frequency domain processing module constitute the current level StageBlock, and the sum of their outputs is the output of the current level StageBlock. Taking I2 as the output of the current level downsampling operation as an example, I2 is input into the two modules for processing. The first module is the frequency domain processing module, which first performs a Fourier transform on I2 to convert it to the frequency domain, and then performs a convolution on I2 in the frequency domain. The operation is as follows:

[0047] I f =ifft(ReLU(fft(I2)*θ2)*θ3) (3)

[0048] Here, fft represents Fourier transform, ifft represents inverse Fourier transform, and ReLU represents activation function; θ2 represents a 3×3 convolution kernel used to expand the dimension of the input feature fft(I2) by incorporating information from the surrounding pixels; θ3 is a 1×1 convolution kernel that aggregates the expanded feature information along the channel dimension. It should be noted that θ2 and θ3 are both frequency domain convolutions, which together reduce the size of the frequency domain image and expand the number of channels, thus achieving the filtering of frequency domain information.

[0049] The second module is the Transformer module, which consists of multiple cascaded Blocks. Figure 5 This is a schematic diagram of the structure of a Block in a Transformer provided by an embodiment of the present invention, as shown below. Figure 5 As shown, I2 is processed using a Block in Transformer as the processing unit to obtain I. t :

[0050] I t =MLP(Norm(Attention(Norm(I2)))) (4)

[0051] Here, Attention represents the attention mechanism. The attention mechanism of the Transformer module can obtain global relevance, represented as... Norm represents the normalization layer, and its formula is as follows: MLP stands for Multilayer Perceptron.

[0052] After multiple levels of block processing, the final output of the Transformer module is denoted as I'. t The outputs of the frequency domain processing module and the Transformer module are added together to obtain the output I3 = I of the current StageBlock. f +I' t .

[0053] S1-3: Use the output of the current StageBlock as the new current image and return to S1-1, repeating this process until all downsampling operations and StageBlocks in the encoder have been completed. Taking I3 as the output of the current StageBlock as an example, use I3 as the new current image, return to S1-1, and repeat this process until all downsampling and StageBlock operations have been completed. Optionally, the number of downsampling operations and StageBlocks should ensure that the output of the last StageBlock is reduced to below 32×32 before being used as the encoder output.

[0054] After obtaining the encoder's output, the input is taken to another StageBlock, independent of the encoder and decoder, for further feature extraction. This StageBlock also includes parallel Transformer modules and frequency domain processing modules. Since there are no upsampling or downsampling operations, the input and output dimensions of this StageBlock are the same. For example, in... Figure 3 In this case, the input and output dimensions of this StageBlock are both 1×32×32×1024.

[0055] After obtaining the output of the next StageBlock, it is used as the input to the decoder. Multiple levels of upsampling and feature extraction are alternately performed within the decoder to obtain the deblurred image. Each StageBlock in the decoder also includes a frequency domain processing module and a Transformer module, with operations consistent with those in the encoder; however, each upsampling operation in the decoder is implemented through a pixel shuffle operation. In one specific embodiment, the output of the first StageBlock is used as the current image input to the decoder; operations S2-1, S2-2, and S2-3 are performed in the decoder:

[0056] S2-1. Perform a Pixel Shuffle operation on the current image to expand the image size and reduce the channel dimension, wherein the Pixel Shuffle operation constitutes the current first-level upsampling operation.

[0057] S2-2. The output of the Pixel Shuffle operation is input into the Transformer module and the frequency domain processing module respectively to extract spatial domain features and filter frequency domain information. The Transformer module and the frequency domain processing module constitute the current stage block. The output of the current stage block is obtained by adding the outputs of the Transformer module and the frequency domain processing module.

[0058] S2-3. Add the output of the current StageBlock to the input of the first StageBlock in the encoder to obtain the new current image, wherein the output of the current StageBlock and the input of the first StageBlock in the encoder have the same dimension; return to the step of performing Pixel Shuffle operation on the current image until each upsampling operation and StageBlock in the decoder has been completed; use the final current image as the deblurred image.

[0059] Step 2: After obtaining the deblurred image, input both the deblurred image and its corresponding clear image into the discriminator of the GAN model for feature extraction. This step inputs the generator's output and its corresponding clear image into the discriminator for feature extraction. The extracted features are used to compare similarities and calculate the model loss. Optionally, the discriminator uses PatchGAN. Compared to the native GAN discriminator which only outputs a single value, PatchGAN is designed as a fully convolutional network, mapping the input to an N×N feature map. Each value represents an evaluation of a region in the original image. That is, inputting the deblurred image into the PatchGAN discriminator for feature extraction yields an N×N first feature map; inputting the clear image corresponding to the blurred image into the PatchGAN discriminator for feature extraction yields an N×N second feature map. PatchGAN considers the influence of different parts of the image, much like considering multiple opinions before making a decision. For image domains requiring high resolution and detailed images, ordinary GAN discriminators are not suitable. Therefore, PatchGAN is used. Its receptive field corresponds to a small region in the input image, which corresponds to the discriminator's discriminative output for that small region of the input image. This training method allows the model to focus more on image details. In one specific implementation, PatchGAN uses three convolutional layers, where the i-th convolutional layer is represented as:

[0060] outputi=ReLU(InstanceNorm(output i-1 *θ i (5)

[0061] Where output0 = I3.

[0062] Step 3: Calculate the model loss based on the extracted features, and continuously update the parameters of the generator and discriminator based on the calculation results. Repeat this process until the GAN model reaches Nash equilibrium. In Nash equilibrium, the features extracted from the deblurred image and the sharp image after passing through the discriminator tend to be consistent. Optionally, the native GAN loss function can be used to calculate the model loss.

[0063]

[0064] Here, D represents the discriminator, and G represents the generator. The training process involves first training the discriminator D with the parameters of the generator G fixed, maximizing the values ​​of D(x) and log(1-D(G(z))). After training the discriminator D, the parameters of the discriminator D are fixed, and the generator G is trained. Overall, this involves first maximizing the loss function from the perspective of the discriminator D, and then minimizing the loss function from the perspective of the generator G. Then, the max problem is transformed into a min problem, resulting in:

[0065]

[0066]

[0067] The converted form is BCELoss: Among them, y n It is the label value (i.e., the value of each element in the second feature map), x n The input values ​​are obtained from the generated image (i.e., the values ​​of each element in the first feature map). Combined with the PatchGAN discriminator, the generator output is convolved with the clear image each time to obtain a feature map. Each element represents an evaluation metric for a region, which is then input into BECLoss to obtain the final loss result. After calculating the loss, the parameters of the generator and discriminator are updated based on the backpropagation results. The GAN model iterates and updates its parameters continuously through a game-like interaction between the generator and discriminator, repeating steps one through three until the network finally reaches Nash equilibrium.

[0068] S130. Extract the trained generator from the trained GAN model, and use the trained generator to deblur the image that needs to be deblurred.

[0069] After reaching Nash equilibrium, the generator is extracted from the trained GAN model, training parameters are loaded, and it is put into use. When image deblurring is needed, the image to be deblurred is input into the generator, and a clear image is output. After processing the blurred image using the method of this embodiment, the blurred image and the deblurred image are compared in the spatial and frequency domains, such as... Figure 6 As shown, from the spatial domain perspective, the deblurred image restores clearly visible image details; from the frequency domain perspective, the deblurred image removes the striped frequency domain components caused by motion blur, achieving a better deblurring effect.

[0070] It should be noted that both the generator and discriminator in this application perform deep feature extraction on the input data, but the features extracted from each layer are different. In the generator, multi-layer deep feature extraction is performed through the encoder to better represent the semantics of the image; as the network depth increases, the feature depth also increases. Then, the decoder continues to extract features while restoring the image size to obtain clearer image content. The feature extraction in the discriminator is similar to that of the encoder in the generator; it only needs to extract depth features to reflect the image category and does not need to restore the image size.

[0071] This invention proposes a method for removing motion blur from images using an end-to-end GAN model. This method restores images blurred by factors such as object motion, camera shake, defocusing, and atmospheric turbulence, resulting in clear and recognizable images. This method eliminates the need to estimate the blur kernel, reducing computational complexity, and can handle uneven motion blur, broadening its application scope. Specifically, this method combines a GAN model with a Transformer. The GAN model only uses backpropagation, eliminating the need for complex Markov chains, and can generate the most realistic images among various generative models, effectively restoring image texture details. The Transformer, on the other hand, has powerful modeling capabilities, and its performance improves with larger datasets. Increased datasets themselves enhance the accuracy of model simulation, thus significantly improving the deblurring effect. Furthermore, in this embodiment, StageBlock extracts spatial features using the Transformer module, filters frequency domain information using a frequency domain processing module, and combines the outputs of the Transformer module and the frequency domain processing module for image deblurring. Since deblurring in the spatial domain can easily lose image detail information, while the frequency domain can preserve image texture details, but the image edges in the frequency domain are prone to oversmoothing, this embodiment combines the spatial and frequency domains to deblur the image, which can balance the advantages and disadvantages of spatial and frequency domain processing, compensate for each other's shortcomings, and improve the final deblurring effect.

[0072] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 7 As shown, the device includes a processor 50, a memory 51, an input device 52, and an output device 53; the number of processors 50 in the device can be one or more. Figure 6 Taking a processor 50 as an example; the processor 50, memory 51, input device 52, and output device 53 in the device can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0073] The memory 51, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the GAN-based image motion blur removal method in this embodiment of the invention. The processor 50 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 51, thereby implementing the aforementioned GAN-based image motion blur removal method.

[0074] The memory 51 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 51 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 51 may further include memory remotely located relative to the processor 50, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0075] Input device 52 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 53 may include display devices such as a display screen.

[0076] This invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the GAN-based method for removing motion blur from images according to any embodiment.

[0077] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0078] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0079] The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. The computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for removing image motion blur based on GAN, characterized in that, The method comprises: obtaining a training set composed of multiple blurred images and corresponding clear images, wherein the multiple blurred images include non-uniform motion blurred images; training a generator and a discriminator of a GAN model based on the training set, so that the features of the blurred images obtained after passing through the generator and the discriminator are consistent with the features of the corresponding clear images obtained after passing through the discriminator; extracting the trained generator from the trained GAN model, and using the trained generator to deblur images that need to be deblurred; wherein the generator adopts a U-net structure, the U-net structure includes a symmetrical encoder and a decoder, the encoder includes alternately distributed multi-level down-sampling operations and StageBlocks, and the decoder includes alternately distributed multi-level up-sampling operations and StageBlocks; each level of the StageBlock in the encoder and each level of the StageBlock in the decoder adopt residual connection; each level of the StageBlock includes a parallel Transformer module and a frequency domain processing module, the Transformer module is used to extract spatial domain features of the image, the frequency domain processing module is used to filter frequency domain information of the image, and the outputs of the Transformer module and the frequency domain processing module are added as the output of each level of the StageBlock; in the training, any blurred image in the training set is input into the encoder, multi-level down-sampling and feature extraction are alternately performed in the encoder, the output of the encoder is input into another level of the StageBlock which is independent of the encoder and the decoder, feature extraction is continued, the output of the another level of the StageBlock is input into the decoder, multi-level up-sampling and feature extraction are alternately performed in the decoder, and a deblurred image is obtained; the deblurred image and the clear image corresponding to the blurred image are input into the discriminator of the GAN model for feature extraction; the model loss is calculated according to the extracted features, and the parameters of the generator and the discriminator are updated according to the calculation result, and the cycle is repeated until the GAN model reaches Nash equilibrium; wherein in the Nash equilibrium, the features extracted by the discriminator after the deblurred image and the clear image pass through the discriminator tend to be consistent.

2. The method of claim 1, wherein, The method comprises: obtaining multiple original blurred images and clear images corresponding to each original blurred image; cutting and recombining each original blurred image and clear image to form a training set from the cut images.

3. The method of claim 1, wherein, The method comprises: performing convolution operation on any blurred image in the training set to reduce the image size and expand the channel dimension; inputting the image after the convolution operation into the encoder as a current image, and performing the following operations in the encoder: S1-1, performing inverse operation of Pixel Shuffle on the current image to continue reducing the image size and expanding the channel dimension, wherein the inverse operation of Pixel Shuffle constitutes the current level down-sampling operation; S1-2, inputting the output of the inverse operation of Pixel Shuffle into a Transformer module and a frequency domain processing module respectively to extract spatial domain features and screen frequency domain information respectively, wherein the Transformer module and the frequency domain processing module constitute the current level StageBlock, and the output of the Transformer module and the frequency domain processing module after being added is the output of the current level StageBlock; S1-3, taking the output of the current level StageBlock as a new current image and returning to S1-1 until each level down-sampling operation and StageBlock in the encoder is performed.

4. The method of claim 1, wherein, The output of the level StageBlock is taken as the input of the decoder, and multi-level up-sampling and feature extraction are alternately performed in the decoder to obtain a deblurred image, comprising: taking the output of the level StageBlock as the current image input into the decoder; in the decoder, the following operations are performed: S2-1, performing Pixel Shuffle operation on the current image to expand the image size and reduce the channel dimension, wherein the Pixel Shuffle operation constitutes the current level up-sampling operation; S2-2, inputting the output of the Pixel Shuffle operation into a Transformer module and a frequency domain processing module respectively to extract spatial domain features and screen frequency domain information respectively, wherein the Transformer module and the frequency domain processing module constitute the current level StageBlock, and the output of the Transformer module and the frequency domain processing module after being added is the output of the current level StageBlock; S2-3, adding the output of the current level StageBlock to the input of the level StageBlock in the encoder as a new current image, wherein the output of the current level StageBlock and the input of the level StageBlock in the encoder have the same dimension; returning to the step of performing Pixel Shuffle operation on the current image until each level up-sampling operation and StageBlock in the decoder is performed; and taking the final current image as a deblurred image.

5. The method of claim 3, wherein, The output of the inverse operation of Pixel Shuffle is input into a Transformer module and a frequency domain processing module respectively to extract spatial domain features and screen frequency domain information respectively, comprising: taking the output of the inverse operation of Pixel Shuffle as the input of the Transformer module, and in the frequency domain processing module, the following operations are performed: S3-1, performing Fourier transform on the output of the inverse operation of the Pixel Shuffle to obtain a frequency domain image; S3-2, performing frequency domain convolution on the frequency domain image to reduce the frequency domain size and expand the frequency domain channel to screen the frequency domain information; S3-3, performing inverse Fourier transform on the output of the frequency domain convolution to transform back to the spatial domain, and taking the transformed result as the output of the frequency domain processing module.

6. The method of claim 1, wherein, The deblurred image and the clear image corresponding to the blurred image are respectively input into a discriminator of the GAN model for feature extraction, including: The deblurred image is input into a discriminator of PatchGAN for feature extraction to obtain a first feature map; The clear image corresponding to the blurred image is input into a discriminator of PatchGAN for feature extraction to obtain a second feature map; Each element value in each feature map represents the evaluation of a region in the original image.

7. An electronic device, comprising: It includes: One or more processors; Memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the GAN-based image motion blur removal method as claimed in any one of claims 1-6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the GAN-based image motion blur removal method as claimed in any one of claims 1-6.

Images