An image inpainting method based on frequency domain convolution and large convolution

Abstract

The application discloses an image restoration method based on frequency domain convolution and large convolution, which can be used to realize complete restoration of a damaged image, maintain continuity of texture and rationality of semantics, and conform to human perception. The method comprises the following steps: obtaining a to-be-restored image blocked by a mask through preprocessing according to original image data and an image mask, and constructing a training data set; constructing an image restoration LKFFC-GAN generative adversarial network model based on frequency domain convolution and large convolution; inputting the constructed training data set into a generator of the LKFFC-GAN network and outputting a restored image; inputting the restored image into a discriminator of the LKFFC-GAN network to obtain the probability that each part of the image is discriminated as an original image, and jointly training the generator and the discriminator through a weighted loss function; and evaluating the training effect of the model LKFFC-GAN on a test set. Compared with current mainstream models, the application achieves better restoration effect in L1 error, peak signal-to-noise ratio (PSNR), and structural similarity index (SSIM).

Description

Technical Field

[0001] This invention relates to an image inpainting method based on frequency domain convolution and large convolution, applicable to the field of image inpainting technology in computer vision. Background Technology

[0002] Image restoration technology refers to filling in missing pixels in an image using contextual information and existing texture information to create a complete image. Ideally, the image obtained through image restoration should be identical to the original. However, since the essence of image restoration is to infer unknown information from known information, there is no single solution in practical research. This is one of the challenges of image restoration. The larger the missing portion of the image, the less usable information there is, and the more possible restoration results there are. Currently, the development of computer vision has facilitated the emergence of digital image restoration. It does not alter the original image data or artifacts, offers high flexibility, and is relatively safe. Furthermore, the application of digital image restoration is not limited to artifact restoration; when people's photographs are damaged or contaminated with dye, digital image restoration technology can be used to restore them, which has practical significance for people's daily lives and work.

[0003] In 2014, Goodfellow et al. first proposed Generative Adversarial Networks (GANs) to address the generative model problem. The generative model problem involves inputting samples into a generative model to learn the distribution of those samples, with the ultimate goal of generating new samples with the same distribution. Before the advent of GANs, deep learning-based generative models had already achieved great success. However, the emergence of GANs, especially their success in generating high-resolution images, has garnered increasing attention. Image inpainting methods based on GANs have also emerged. Context Encoders models, influenced by GANs during training, incorporate the adversarial concept into the training process, making them essentially a type of GAN. Iizuka et al. proposed a GAN based on a fully convolutional neural network, utilizing local and global context discriminators to distinguish real images. The global discriminator observes the entire image, judging its consistency as a whole, while the local discriminator only observes local patches of the image, ensuring local consistency. Wang et al. applied Conditional Generative Adversarial Networks (GANs) to image inpainting, designing a set of multi-scale generators and discriminators that integrate large-scale and small-scale features, thus solving the problem of GANs being limited by low resolution. Ding et al. proposed using depthwise separable convolutions with large kernels to replace multi-layer convolutions with small kernels, which reduced the complexity of the model while maintaining a large receptive field, achieving good results in downstream computer vision tasks. Summary of the Invention

[0004] Purpose of the invention: To address the existing difficulties in image restoration tasks, this invention proposes an image restoration method based on frequency domain convolution and large convolution, which improves the quality of restored images.

[0005] Technical solution: An image inpainting method based on frequency domain convolution and large convolution, including the following steps:

[0006] Step S1: Construct an image dataset for image inpainting, randomly selecting image P from the source dataset. real From the mask dataset, mask N is used for image preprocessing to generate the image P to be repaired. masked ;

[0007] Step S2: Construct the generator and discriminator of the image inpainting generative adversarial network LKFFC-GAN;

[0008] Step S3: Take the P obtained in step S1 masked The generator of the LKFFC-GAN network is fed with N data and forward processed to obtain the generator's output I. G Using I GThe final repaired image P is obtained from N. comp The loss function of the generator is calculated, and the Adam optimizer is used to optimize and update the generator of the LKFFC-GAN network. The process is iterated to obtain the optimized generator of the LKFFC-GAN network.

[0009] Step S4: Transfer the image P obtained in step S3 to... comp The input is fed into the discriminator of the LKFFC-GAN network for forward computation to obtain the discriminator's decision P. comp Each part represents the probability of the original image. The loss function of the discriminator is calculated, and the Adam optimizer is used to optimize and update the discriminator of the LKFFC-GAN network, thus obtaining the optimized discriminator of the LKFFC-GAN network.

[0010] Step S5: After training the generator and discriminator of the LKFFC-GAN network through steps S3 and S4, input the test set data into the generator of the LKFFC-GAN network according to steps S1 and S3 to obtain the repaired image.

[0011] Furthermore, step S1 specifically includes:

[0012] Step S101: Randomly select a mask N from the image mask dataset and reduce its size to 256×256. Similarly, randomly select a real image P from the image dataset. real The size was reduced to 256×256;

[0013] Step S102: For P real Perform data augmentation operations such as random rotation, random cropping and scaling;

[0014] Step S103: The mask N obtained in step S101 and the real image P real The image P to be repaired is calculated using equation (1). masked .

[0015] P masked = (1-N)×P real +N (1)

[0016] Furthermore, step S3 specifically includes:

[0017] Step S301: Combine the image mask N obtained in step S101 with the image P to be repaired obtained in step S103. masked The LKFFC-GAN network generator is trained by inputting the LKFFC-Block module to extract global information in the frequency domain and then extract multi-scale information for feature aggregation.

[0018] Step S302: The LKFFC-GAN network generator is trained unsupervised using L1, Style, Perceptual, and generator adversarial loss functions. During training, training stops once the loss function values ​​converge, thus obtaining the optimized LKFFC-GAN generative adversarial network generator and obtaining the generator output I. G ;

[0019] Step S303: The generator output I obtained in step S302 G The image mask N obtained in step S101 can be used to obtain the final repaired image P from equation (2). comp .

[0020] P comp = (1-N)×P real +N×I G (2)

[0021] Furthermore, step S4 specifically includes:

[0022] Step S401: The final repaired image P obtained in step S303 is... comp It is used as input to the discriminator of the LKFFC-GAN generative adversarial network for training;

[0023] Step S402: Unsupervised training of the LKFFC-GAN network discriminator is performed using the discriminator adversarial loss function. During training, training stops once the loss function value converges, resulting in an optimized and usable LKFFC-GAN generative adversarial network discriminator. The discriminator outputs a decision P. comp Each part is the probability matrix F of the original image.

[0024] Furthermore, step S5 specifically includes:

[0025] Step S501: Fix the LKFFC-GAN network generator parameters obtained by training in step S302;

[0026] Step S502: Using the test set image and the image mask as input, obtain the final repaired image P of the test set using equation (2). comp .

[0027] The beneficial effects of this invention are: it can complete high-quality restoration of damaged images, maintaining the consistency of texture and the rationality of semantics after restoration. Firstly, it uses the original image P... real The image P to be repaired is obtained by using the image mask N. masked Then P masked Using N as input to the LKFFC-GAN network generator, the repaired image P is obtained through calculation. compCompared with the comparison algorithm, the method proposed in this invention achieves better results in terms of L1 error, peak signal-to-noise ratio, and structural similarity of the repaired image, as tested. This indicates that the method of this invention can more effectively and with higher quality complete image restoration work. From a qualitative perspective, the restored image obtained by the method of this invention is more in line with human cognition, indicating that it can better restore the semantic information of the image to be restored. Furthermore, the frequency domain convolution technique and large convolution technique used provide feasible solutions for further exploration by future researchers and have further reference value. Attached Figure Description

[0028] Figure 1 This is a flowchart of the image inpainting method based on frequency domain convolution and large convolution in Example 1.

[0029] Figure 2 This is a flowchart of the training and prediction process of the LKFFC-GAN generative adversarial network in Example 1.

[0030] Figure 3 This is a structural diagram of the LKFFC-GAN generative adversarial network model in Example 1.

[0031] Figure 4 This is a structural diagram of the LKFFC-Block module in Example 1.

[0032] Figure 5 This is a diagram showing the effect of image restoration in Example 1. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Example 1

[0035] See Figures 1-5 This invention can perform high-quality restoration of damaged images. First, it uses the original image P... real The image P to be repaired is obtained by using the image mask N. masked Then P masked Using N as input to the LKFFC-GAN network generator, the repaired image P is obtained through calculation. comp The repaired image is used as input to the LKFFC-GAN network discriminator, which outputs the probability that each part is a real image.

[0036] Specifically, see Figure 1 This method specifically includes:

[0037] Step S1: Construct an image dataset for image inpainting, and randomly select image P from the dataset. real From the mask dataset, mask N is used for image preprocessing to generate the image P to be repaired. masked ;

[0038] More specifically, step S1 includes: randomly selecting a mask N from the image mask dataset and reducing its size to 256×256. Similarly, randomly selecting a real image P from the image dataset... real The size is reduced to 256×256, and data augmentation operations such as random rotation, random cropping, and scaling are performed; the mask N and the real image P are used. real The image P to be repaired is calculated. masked .

[0039] In this example, the image mask N is a single-channel grayscale image. A value of 1 (white) represents a portion of the image that is corrupted and needs to be restored, while a value of 0 (black) represents a portion of the image that remains consistent with the original image. To ensure the robustness of the training results, this example uses random rotation, random cropping, and scaling for data augmentation. The random selection of the mask in this example guarantees the randomness of image corruption.

[0040] Step S2: Construct the generator and discriminator of the image inpainting generative adversarial network LKFFC-GAN;

[0041] Step S3: Take the P obtained in step S1 masked The generator of the LKFFC-GAN network is fed with N data and forward processed to obtain the generator's output I. G Using I G The image P obtained by N is the final repair image. comp The loss function of the generator is calculated, and the Adam optimizer is used to optimize and update the generator of the LKFFC-GAN network. The process is iterated to obtain the optimized generator of the LKFFC-GAN network.

[0042] More specifically, in this embodiment, step S3 includes: using the image mask N obtained in step S1 and the image to be repaired as inputs to the LKFFC-GAN network generator for training; performing unsupervised training of the LKFFC-GAN network generator using L1, Style, Perceptual, and generator adversarial loss functions; stopping training after the loss function values ​​converge during training, thereby obtaining an optimized and usable LKFFC-GAN generative adversarial network generator, and obtaining the generator output I. G The generator outputs I. GThe final repaired image P can be obtained by calculating the image mask N. comp .

[0043] Step S4: Transfer the image P obtained in step S3 to... comp The input is fed into the discriminator of the LKFFC-GAN network for forward computation to obtain the discriminator's decision P. comp Each part represents the probability of the original image. The loss function of the discriminator is calculated, and the Adam optimizer is used to optimize and update the discriminator of the LKFFC-GAN network. This process is iterated to obtain the optimized discriminator of the LKFFC-GAN network.

[0044] More specifically, in this embodiment, the final repaired image P obtained in step S3 is... comp The LKFFC-GAN generative adversarial network discriminator is trained by inputting the discriminator adversarial loss function. The discriminator adversarial loss function is used to train the LKFFC-GAN discriminator in an unsupervised manner. During the training process, the training stops after the loss function value converges, thus obtaining the optimized and usable LKFFC-GAN generative adversarial network discriminator.

[0045] It should be noted that in this embodiment, the training of the LKFFC-GAN network generator in step S3 and the training of the LKFFC-GAN network discriminator in step S4 are carried out alternately, that is, the two are competing with each other and moving closer to the equilibrium point where the generator produces new samples with the same distribution as the real data.

[0046] Step S5: After training the generator and discriminator of the LKFFC-GAN network through steps S3 and S4, input the test set data into the generator of the LKFFC-GAN network according to steps S1 and S3 to obtain the repaired image.

[0047] More specifically, in this embodiment, the LKFFC-GAN network generator parameters obtained by training in step S302 are fixed; the test set image and the image mask are used as input, and the final repaired image of the test set is obtained by equation (2).

[0048] like Figure 3 As shown, this model is the overall structure of the LKFFC-GAN generative adversarial network. In the generator, four LKFFC-Block modules are connected serially. The input data pairs include the image P to be repaired. masked And the image mask N. The input data first passes through the encoder module of the generator to extract shallow features, then through the LKFFC-Block module to extract the image frequency domain information, and then uses convolutional kernels of different sizes to extract multi-scale information before feature aggregation, which is then input into the decoder of the generator, and finally by the output I of the generator. G The repaired image is calculated using the image mask N and used as input to the discriminator.

[0049] like Figure 4 As shown, this model represents the internal structure of the LKFFC-Block module. Each LKFFC-Block module receives a 256-channel tensor as input. Internally, the 256 channels are first divided into two groups of 128 channels each. The first group undergoes a Fast Fourier Transform (FFT) followed by a 2D convolutional module to extract frequency domain information. Then, an Inverse Fast Fourier Transform (IFFT) is performed to recover the time domain information. The second group of 128 channels is input to a 3-kernel 2D convolutional layer to extract local information. The two outputs are then aggregated along the channel dimension to obtain a 256-channel output. This output is further processed through multiple large convolutional modules with varying kernel sizes to extract multi-scale information. Finally, the output of the LKFFC-Block module is obtained via a gated residual module.

[0050] like Figure 5 The image shown is a partial result of using the LKFFC-GAN generative adversarial network for image inpainting. The image mask dataset is divided into six parts according to the occlusion range: occlusion rate 0%-10%, 10%-20%, 20%-30%, 30%-40%, 40%-50%, and 50%-60%. Qualitatively, the LKFFC-GAN generative adversarial network achieves better results than the control network in restoring both image texture and semantics. This is especially true when the image to be repaired has large areas of damage. As shown in Table 1, quantitatively, LKFFC-GAN achieves better results in all three selected metrics: L1 error, PSNR, and SSIM. This is consistent with the results obtained from qualitative observations.

[0051] Table 1 shows the L1 error, peak signal-to-noise ratio, and structural similarity values ​​in Example 1.

[0052]

[0053] Any aspects of this invention not described in detail are well-known to those skilled in the art.

[0054] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An image inpainting method based on frequency domain convolution and large convolution, characterized in that, The method includes the following steps: Step S1: constructing an image dataset for image inpainting, randomly selecting an image in the dataset and a mask in the mask dataset , performing image preprocessing thereon to generate an image to be inpainted ; Step S2: Construct an image inpainting network LKFFC-GAN based on generative adversarial networks, which includes a generator and a discriminator; Step S3: Obtain the information from step S1 and The data is fed into the generator of the LKFFC-GAN network for forward propagation to obtain the generator's output. ,use and Obtain the final restored image ; Calculate the loss function of the generator, and use the Adam optimizer to optimize and update the generator of the LKFFC-GAN network, and iterate to obtain the optimized generator of the LKFFC-GAN network. Step S4: Transfer the image obtained in step S3 The input is fed into the discriminator of the LKFFC-GAN network for forward computation to obtain the discriminator's decision. The probability that each part is the original image; The loss function of the discriminator is calculated, and the Adam optimizer is used to optimize and update the discriminator of the LKFFC-GAN network, thereby obtaining the optimized discriminator of the LKFFC-GAN network. Step S5: The generator and discriminator of the LKFFC-GAN network are trained through steps S3 and S4. The test set data is input into the generator of the LKFFC-GAN network according to steps S1 and S3 to obtain the repaired image. Step S1 specifically includes: Step S101: Randomly select an image mask from the image mask dataset. Reduce the size to 256 256; Randomly select real images from the image dataset. Reduce the size to 256 256; Step S102: For Perform data enhancement operations such as random rotation and random cropping / scaling; Step S103: obtaining the mask from step S101 and the real image calculating the image to be repaired from the real image by formula (1) , （1）； Step S3 specifically includes: Step S301: Apply the image mask obtained in step S101 The image to be repaired obtained in step S103 The LKFFC-GAN network generator is trained by inputting the LKFFC-Block module to extract global information in the frequency domain and then extract multi-scale information for feature aggregation. Step S302: Using The LKFFC-GAN network generator is trained unsupervised using the Style, Perceptual, and Generator Adversarial loss functions. Training stops once the loss function values ​​converge, resulting in an optimized and usable LKFFC-GAN generative adversarial network generator, from which the generator output is obtained. ; Step S303: Generator output obtained in step S302 and the image mask obtained in step S101 The final restored image is obtained from equation (2). , （2）； Step S4 specifically includes: Step S401: The final repaired image obtained in step S303 It is used as input to the discriminator of the LKFFC-GAN generative adversarial network for training; Step S402: Unsupervised training of the LKFFC-GAN network discriminator is performed using the discriminator adversarial loss function. During training, training stops once the loss function value converges, resulting in an optimized and usable LKFFC-GAN generative adversarial network discriminator; the discriminator outputs a judgment. Each part is a probability matrix of the original image. ; Step S5 specifically includes: Step S501: Fix the LKFFC-GAN network generator parameters obtained by training in step S302; Step S502: taking the test set image and the image mask as inputs, obtaining the final repaired image of the test set by formula (2) .

Images