A Few-Shot Image Inpainting Method Based on Iterative Residual Learning

Abstract

The present invention proposes a few-shot image inpainting method based on iterative residual learning, which includes a generator network, a mapping space discriminator, and a forged patch discriminator network. In each training, the model will iterate multiple times and optimize the inpainting results. In each iterative optimization, the generator will reuse the previous inpainting results and further predict the residual information between the inpainted area and the original sample. The new round of inpainting results is obtained by combining the residual information, and the loss value is calculated for it. The loss value is calculated by combining the mapping space generative adversarial loss, the forged patch generative adversarial loss, and the perceptual loss, and backpropagation is performed to adjust the parameters of the generator network, the mapping space discriminator, and the forged patch discriminator network. Repeat the above steps until the training ends, and select the optimal network parameters as the inpainting model parameters; implementing the present invention can achieve an image inpainting method that achieves high-quality inpainting on few-shot and multi-shot data sets.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular, to a few-shot image restoration method based on iterative residual learning. Background Art

[0002] In image restoration methods, early diffusion-based methods and patch-based methods mainly utilize similar information around the image to fill in the missing regions of the image. These methods perform well in the case of small missing regions. However, due to the inability to perceive the global semantic relationship of the image, the performance of the model is not good in the case of large missing regions.

[0003] Existing deep learning-based restoration methods can generate satisfactory results in the missing regions, but these methods have the following problems: First, these methods rely on a large number of training samples, and limited small samples are likely to cause overfitting of the trained model. Second, the problem of overfitting can be alleviated by model lightweighting, but lightweight models have limited learning ability and cannot still guarantee good performance on large sample datasets. The above two problems further limit the use of image restoration in real-world scenarios. For example, medical images, art images, and historical relic images are difficult to collect. It is difficult to achieve good performance in these image domains using existing restoration model methods. Summary of the Invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide a few-shot image restoration method based on iterative residual learning, which can not only achieve good restoration results on large datasets, but also achieve excellent restoration performance on few-shot datasets.

[0005] To solve the above technical problem, the embodiments of the present invention provide a few-shot image restoration method based on iterative residual learning, which constructs an image restoration model to perform restoration processing on the input image. The characteristic is that the image restoration model is obtained at least through the following steps:

[0006] Step S1: Given a training set and a test set where, \(I\) gti represents the \(i\)-th image in the training set, 3 represents the number of color channels, and the corresponding color channel \(d\in\{red, green, blue\}\), \(K\) represents the size of the width or height of a single image; represents the \(j\)-th image in the test set. \(N\) represents the number of samples in the training set \(I\) train ; \(H\) represents the number of samples in the test set \(I\) test ;

[0007] Step S2: Construct a network model for image inpainting based on iterative residuals. The network model includes at least a generator network, a spatial mapping discriminator, and a forged patch discriminator. Among them, a convolutional neural network and a Transformer are used to construct the generator network G(·), which is used to predict the residual information between the inpainted region of the current input and the real sample by combining the output result optimized in the previous iteration, and obtain a new round of inpainted result. Construct a spatial mapping discriminator D(·) for generating adversarial training in the mapping space of the pre-trained model to assist the generator G(·) in learning the feature distribution of real samples in the mapping space of the classifier network. The parameters of D(·) are denoted as θ d ; Construct a forged patch discriminator E(·) for generating adversarial training on image patches and image details to assist the generator G(·) in learning the detailed features of real samples. The parameters of E(·) are denoted as θ e ; Finally, initialize the models of all networks. Therefore, during the q-th training, the parameters of the generator network G(·), the spatial mapping discriminator D(·), and the forged patch discriminator E(·) are set to

[0008] Step S3: During the q-th training process, traverse all samples in the training set. For the currently obtained i-th training sample Generate a random binary mask to construct an initial input image and set Set the initial input image as the inpainted result of the 0-th time. In the t-th iterative optimization, by inputting the previous inpainted result and the corresponding binary mask M i into the generator G(·), the generator predicts the corresponding residual information and processes it to obtain a new round of inpainted result

[0009] The loss value is calculated for each iterative optimization. The loss value is calculated by combining the adversarial loss in the mapping space, the adversarial loss of forged patches, and the perceptual loss. According to the loss value, backpropagation is performed, and the parameters of the generator network G(·), the mapping space discriminator D(·), and the forged patch discriminator E(·) are updated and adjusted using the stochastic gradient descent method.

[0010] Step S4: Use the test set I test to evaluate the method model of the present invention. If the inpainting quality index of the model in the test image is the highest currently, save the model parameters, and let and Denote the optimal model parameters of the current generator network G(·), mapping space discriminator D(·), and forged patch discriminator E(·). Determine whether the training has reached the maximum number of training times n. If the maximum number of training times n has been reached, the training phase ends and jumps to step S5; otherwise, it jumps to step S3.

[0011] Step S5: Obtain the model parameters of the generator network G(·), mapping space discriminator D(·), and forged patch discriminator E(·). And use the generator network for image inpainting of iterative residual learning.

[0012] As a further improvement scheme, in the said step S1, the training set I train and the test set I test are both image sets and do not require image labels.

[0013] As a further improvement, in the step S2, the generator G(·) is constructed by a convolutional neural network and a Transformer. The convolutional neural network is used to construct the encoder and decoder for extracting visual features; the self-attention mechanism in the Transformer mainly uses the existing information for reasoning, so the Transformer is mainly used for global semantic feature reasoning in abstract features. For the network structures of the encoder and decoder, please refer to FastGAN (B. Liu, Y. Zhu, K. Song, and A. Elgammal, “Towards faster and stabilized GAN training for high-fidelity few-shot image synthesis,” in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.). For the network structure of the Transformer, please refer to Restormer (S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M. H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in CVPR. Washington: IEEE, 2022.). The mapping space discriminator D(·) is constructed by a convolutional neural network. In the present invention, the discriminator with the largest scale in Projected GAN is used, and other discriminators are removed. The structure please refer to Projected GAN (A. Sauer, K. Chitta, J. M·uller, and A. Geiger, “Projected gans converge faster,” in Advances in Neural Information Processing Systems (NeurIPS), 2021.).The pre-trained classification network model P(·) it uses can directly use the pre-trained EfficientNet (M. Tan and Q. Le, "EfficientNet: Rethinking model scaling for convolutional neural networks," in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09 - 15 Jun 2019, pp. 6105 - 6114.) or the VGG network (K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015.). The network of the forged patch discriminator E(·) can directly use the model parameters of the discriminator in PatchGAN (P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, "Image-to-image translation with conditional adversarial networks," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington: IEEE, 2017, pp. 5967 - 5976).

[0014] As a further improvement scheme, in the step S3, given the i-th training sample Generate a random binary mask to construct the initial input image Input image The construction formula of is as follows:

[0015]

[0016] Among them, represents the preprocessed input image; Mi is the corresponding binary mask, where 1 is used to indicate the defective area and 0 represents the known area, and the known area will not be changed. ⊙ is the Hadamard Product. Through the broadcasting mechanism, in , and M i the information of the pixel points corresponding to the masked area will be erased and represented as 0.

[0017] In the case of t - th iteration optimization, the prediction calculation formula for the corresponding residual information is:

[0018]

[0019] where is the predicted residual information; is the inpainting result after the previous iteration optimization. When t = 1, the corresponding input inpainting image is the pre - processed input image that is

[0020] Using the residual information and the previous inpainting result we can obtain the optimized inpainting result, which can be expressed as:

[0021]

[0022] Among them, in the inpainting area specified by the binary mask, the previous inpainting result and the newly predicted residual information are added, and the known area can directly reuse the original input information without change. The number of iterations T can be set at the beginning of training, such as T = 3.

[0023] As a further improvement scheme, in the step S3, the present invention first proposes a mapping space generative adversarial loss, and its calculation formula is:

[0024]

[0025] To simplify the formula, the subscript i is omitted here. Among them, is the objective function for optimizing the parameters of the mapping space discriminator D(·). When optimizing D(·), the parameters of the generator G(·) are fixed and do not participate in the update; is the objective function for optimizing the parameters of the generator G(·). When optimizing G(·), the parameters of the mapping space discriminator D(·) are fixed and do not participate in the update; represents taking the expectation of the result with the input being I gt , Similarly, ReLU(·) is the rectified linear unit function; P(·) is the feature extractor of the pre-trained classification model mentioned above. The mapping space generative adversarial loss directly extracts visual features using the pre-trained feature encoding. The mapping space discriminator D(·) only needs to focus on learning the distribution of abstract visual features, enabling D(·) to capture the semantic information of the overall image with fewer model parameters and alleviating the occurrence of overfitting in few-shot cases.

[0026] Next, the present invention proposes a forged patch generative adversarial loss for capturing the distribution of local shallow information of an image. This loss function can encourage the generator network to further optimize in terms of repair details. After the input information is input into the forged patch discriminator E(·), E(·) outputs a two-dimensional tensor of size w′×h′. Each point of the two-dimensional tensor corresponds to the confidence of the slice within the slice range corresponding to the receptive field size (N×N) of E(·), that is, to what extent this image slice is considered to be an image slice of a real sample. Before calculating the forged patch generative adversarial loss, first construct the corresponding label map whose scale size is the same as the output of E(·). First, use the receptive field of E(·) to crop the input binary mask and the picture according to the receptive field. The cropped sub-slice R can be obtained a，b and 1 ≤ a ≤ h′, 1 ≤ b ≤ m′. The construction formula of the corresponding label map is:

[0027]

[0028] If is not 0, it means that the corresponding sub-slice R a，b region contains the pixel points generated by the generator G(·), and it can be considered that this slice is a forged slice. Conversely, this slice is considered to be a real slice. So the corresponding X a,b = 1 indicates that the slice under the corresponding receptive field region is a forged slice, and X a,b = 0 is considered to be a real slice.

[0029] The calculation formula of the corresponding forged patch generative adversarial loss is as follows:

[0030]

[0031] Where, to simplify the formula, the subscript i is omitted here. The objective function for constraining the parameters of the forged patch discriminator E(·); is the objective function for constraining the parameters of the generator G(·). X is the constructed label map, and its scale size is the same as The output sizes are the same. Under the constraint of the forged patch generation adversarial loss, in order to be able to distinguish between the image patches containing the generated pixels and the image patches of the real samples. The forged patch discriminator not only needs to learn the distribution of the real pixels, but also needs to consider the consistency between the generated pixels and the surrounding pixels. This provides a more explicit and strict constraint for the discriminator. It helps the discriminator quickly capture the differences between the real image patches and the generated image patches. This also makes the generator G(·) have to consider the situation of the generated pixels and the surrounding ones, rather than simply making the generated pixels be able to approach the distribution of the real samples in terms of pixels, thereby enhancing the visual quality of the model in repairing details.

[0032] Perceptual loss L lpips The formula is as follows:

[0033]

[0034] Among them, F(·) is the feature extractor of the pre-trained classification network model. (For the LPIPS loss, please refer to: R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proc. CVPR, 2018, pp. 586 - 595.).

[0035] By jointly constraining the mapping space generation adversarial loss, the forged patch generation adversarial loss, and the perceptual loss, the formula for the final objective function of the generator is obtained as:

[0036]

[0037] θ g ,θ d ,θ e respectively represent the generator network, the mapping space discriminator network, and the forged patch discriminator network; by jointly training the generator network, the mapping space discriminator network, and the forged patch discriminator network, the formula is as follows:

[0038]

[0039]

[0040]

[0041] That is, first, optimize the parameters in the mapping space discriminator network D(·) and the forged patch discriminator network E(·) separately, and then optimize the parameters in the generator network G(·); during the inference process, the generator network G(·) is optimized through multiple iterations to achieve image inpainting based on residual learning; the mapping space discriminator network and the forged patch discriminator network are used to distinguish true and false samples during training, enabling the model to learn the distribution of real samples through a generative adversarial approach.

[0042] Perform backpropagation according to the loss value, and use the stochastic gradient descent method to update and adjust the model parameters of the generator network G(·), the mapping space discriminator D(·), and the forged patch discriminator E(·) at the q-th time and The specific steps are as follows:

[0043] According to the final objective function, use the chain rule of differentiation to calculate the gradients of the model parameters of the generator network G(·), the mapping space discriminator D(·), and the forged patch discriminator E(·) at the q-th time and and update the corresponding parameters through the stochastic gradient descent method:

[0044]

[0045]

[0046]

[0047] where η represents the learning rate in the hyperparameters; and respectively represent the gradients of the parameters corresponding to the generator network G(·), the mapping space discriminator D(·), and the forged patch discriminator E(·) at the q-th training, obtained through the chain rule of differentiation.

[0048] Among them, in step S4, use the test set I test to evaluate the method model of the present invention, and the specific steps are as follows:

[0049] First, generate a fixed set of binary masks according to the test set Traverse the data. Each time, take a sample from the test set I test and the set of binary masks I mask at the same index subscript, and calculate each sample in the calculation method described in step S3, but do not participate in the loss value and gradient calculation, nor perform backpropagation, and the parameters do not participate in the update. Obtain the corresponding set of inpainting results For the test set Itest and the set of repair results Y test Calculate the Fr′echet inception distance metric. (FID, see: M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “GANs trained by a two - time - scale update rule converge to a local nash equilibrium,” in Proc. NIPS, 2017, pp. 6626 - 6637.). Here, define the optimal model network parameters and The FID metric of is fid best , if fid best > fid q , then let fid best = fid q .

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. Compared with the existing image inpainting methods, the present invention can train an image inpainting model based on a small number of samples, and the inpainting model can ensure the visual quality of the inpainted image, having good value and prospects in practical applications;

[0052] 2. The present invention proposes a few - shot image inpainting method based on iterative residual learning. Based on the generative adversarial network architecture, this method first introduces an inpainting framework of iterative residual learning, enabling the model to only predict the residual information between the inpainted region and the original image content using the previous inpainting results at each stage. Under this training mechanism, the model has better generalization ability, achieving good inpainting effects not only on large - sample data sets but also excellent inpainting quality on few - sample data sets. The present invention introduces a mapping - space generative adversarial loss, using the encoded features of a pre - trained classifier, enabling the mapping - space discriminator to capture the global semantic relationships of images and learn global semantic features without introducing too many learning parameters, realizing the learning of the overall semantic distribution of images. Secondly, the present invention proposes a forged - patch generative adversarial loss. Based on the receptive field of the discriminator of the forged patch, the image is divided into multiple partially overlapping image patches. The forged - patch discriminator network is encouraged not only to distinguish the pixel distributions of the inpainted image and the original image but also to consider the consistency between the generated pixels and the surrounding pixels, giving the discriminator clearer and stricter constraints. This constraint also encourages the generator network to learn the pixel details, structures, and textures of the original samples, thereby enhancing the quality of the inpainted image. Brief Description of the Drawings

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, obtaining other drawings based on these drawings still belongs to the scope of the present invention.

[0054] Figure 1 It is a flowchart of a few-shot image inpainting method based on iterative residual learning provided by an embodiment of the present invention;

[0055] Figure 2 It is an architecture diagram of a few-shot image inpainting method based on iterative residual learning provided by an embodiment of the present invention;

[0056] Figure 3 It is the inpainting visual effect using the method of the present invention. Detailed Embodiments

[0057] To make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings.

[0058] In order to overcome the technical defects existing in the prior art, the present invention proposes a few-shot image inpainting method based on iterative residual learning. Based on the generative adversarial network architecture, this method first introduces an inference method of iterative residual learning. By predicting the residual information between the missing region and the original sample each time, the final inpainting result is continuously optimized. Secondly, a mapping space generative adversarial loss function and a forged patch generative adversarial loss function are proposed. The mapping space discriminator uses a pre-trained classification model to extract the abstract semantic features of the image. Therefore, this model only needs fewer learning parameters to realize the learning of the real sample distribution in the encoding space. In this way, the global features of the image are captured, and the overfitting of the model in the case of a small number of samples is prevented. Each unit in the output of the forged patch discriminator corresponds one-to-one with a certain block patch of the input image. By clearly constraining the forged patch discriminator with this mapping relationship, the discriminator can more clearly learn the distribution difference between the forged patch and the real patch, thereby improving the generation quality of the generator in the image detail texture.

[0059] Adopting the technical solution of the present invention can not only achieve good inpainting results on large datasets, but also achieve excellent inpainting performance on few-shot datasets. Moreover, this method can adjust the inpainting quality of the image by controlling the number of iterative optimizations. The following describes the technical solution of the present invention with specific embodiments.

[0060] Such asFigure 1 As shown in the figure, in an embodiment of the present invention, a flowchart of a few-shot image inpainting method based on iterative residual learning is proposed. The method includes the following steps:

[0061] Step S1, given a training set and a test set wherein, represents the i-th image in the training set, 3 represents the number of color channels, and the corresponding color channels d ∈ {red, green, blue}, and K represents the size of the width or height of a single image; represents the j-th image in the test set. N represents the number of samples in the training set I train ; H represents the number of samples in the test set I test . The training set I train and the test set I test are both image sets and do not require image labels.

[0062] Step S2, construct a network model for image inpainting based on iterative residuals; the network model at least includes a generator network, a spatial mapping discriminator, and a forged patch discriminator; wherein, use a convolutional neural network and a Transformer to construct the generator network G(·), which is used to combine the output result of the previous iteration optimization to predict the residual information between the current input inpainting region and the real sample, and obtain a new round of inpainting result; construct a spatial mapping discriminator D(·), which is used to perform generative adversarial training in the mapping space of the pre-trained model to assist the generator G(·) in learning the feature distribution of real samples in the mapping space, and the parameters of D(·) are represented as θ d ; construct a forged patch discriminator E(·), which is used to perform generative adversarial training on image patches and image details to assist the generator G(·) in learning the detailed features of real samples, and the parameters of E(·) are represented as θ e ; finally, initialize the models of all networks. Therefore, at the q-th training, the parameters of the generator network G(·), the spatial mapping discriminator D(·), and the forged patch discriminator E(·) are respectively set as

[0063] wherein, the construction of the generator G(·) is as Figure 2(as shown in (a)). This network is constructed by convolutional layers, residual down-sampling blocks, and AOT-modules in the encoder part; then, the output of the encoder is fed into the network constructed by the Transformer module for further inference. The self-attention mechanism in the Transformer mainly uses the existing information for inference, so the Transformer is mainly used for global semantic feature inference in the abstract features. During this process, skip connections and skip-layer excitation operations are used to aggregate information of multi-scale feature maps in the decoder. The decoder is constructed by up-sampling blocks, AOT-modules, and convolutional layers. For the residual down-sampling blocks, up-sampling blocks, and skip-layer excitation, please refer to FastGAN (B. Liu, Y. Zhu, K. Song, and A. Elgammal, “Towards faster and stabilized GAN training for high-fidelity few-shot image synthesis,” in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.); for the AOT-module, please refer to AOT-GAN (Y. Zeng, J. Fu, H. Chao, and B. Guo, “Aggregated contextual transformations for high-resolution image inpainting,” in Arxiv, 2020.); for the network structure of the Transformer, please refer to Restormer (S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M. H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in CVPR. Washington: IEEE, 2022.).

[0064] The mapping space discriminator D(·) is constructed by a convolutional neural network. In the present invention, the discriminator of the largest scale is used. For the structure, please refer to Projected GAN (A. Sauer, K. Chitta, J. Müller, and A. Geiger, “Projected gans converge faster,” in Advances in Neural Information Processing Systems (NeurIPS), 2021.). The pre-trained classification network model P(·) it uses can directly use the pre-trained EfficientNet (M. Tan and Q. Le, “EfficientNet: Rethinking model scaling for convolutional neural networks,” in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 6105–6114.) or the VGG network (K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015.). The network of the forged patch discriminator E(·) can directly use the model parameters of the discriminator in PatchGAN (P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington: IEEE, 2017, pp. 5967-5976).

[0065] Step S3: During the q-th training process, traverse all the samples in the training set. For the currently obtained i-th training sample Generate a random binary mask to construct the initial input image and set the initial input image as the repair result of the 0-th time. In the t-th iterative optimization, by using the binary mask M corresponding to the previous repair result i input into the generator G(·), the generator predicts the corresponding residual information Δ t and processes it to obtain a new round of repair result

[0066] In each iterative optimization, the loss value will be calculated. The loss value is calculated by combining the mapping space generative adversarial loss, the forged patch generative adversarial loss, and the perceptual loss. According to the loss value, backpropagation is performed, and the parameters of the generator network G(·), the mapping space discriminator D(·), and the forged patch discriminator E(·) are updated and adjusted using the stochastic gradient descent method.

[0067] The input image is constructed as follows:[[]]

[0068]

[0069] where represents the preprocessed input image; M i is the corresponding binary mask, 1 is used to indicate the defective area, 0 represents the known area, and the known area will not be changed. ⊙ is the Hadamard Product. Through the broadcasting mechanism, in , the information of the pixel points corresponding to the covered area in M i will be erased and represented as 0.

[0070] In the case of the t-th iterative optimization, the prediction calculation formula of the corresponding residual information is:[[]]

[0071]

[0072] where is the predicted residual information; is the repair result after the previous iterative optimization. When t = 1, the corresponding input repair image is the preprocessed input image that is Using the residual information and the previous repair result We can obtain the optimized repair result, which can be expressed as:

[0073]

[0074] Among them, in the repair area specified by the binary mask, the previous repair result and the residual information predicted this time are added together, and the known area can directly reuse the original input information without change. The number of iterations T can be set at the beginning of training, such as T = 3.

[0075] As a further improvement scheme, in the step S3, the present invention first proposes a mapping space generative adversarial loss, and its calculation formula is:

[0076]

[0077] To simplify the formula, the subscript i is omitted here. Among them, is the objective function for optimizing the parameters of the mapping space discriminator D(·). When optimizing D(·), the parameters of the generator G(·) are fixed and do not participate in the update; is the objective function for optimizing the parameters of the generator G(·). When optimizing G(·), the parameters of the mapping space discriminator D(·) are fixed and do not participate in the update; represents taking the expected value of the result with the input being I gt ; Similarly; ReLU(·) is the rectified linear unit function; P(·) is the feature extractor of the pre-trained classification model mentioned above. The mapping space generative adversarial loss directly extracts visual features using the pre-trained feature encoding. The mapping space discriminator D(·) only needs to focus on learning the distribution of abstract visual features, enabling D(·) to capture the semantic information of the overall image with fewer model parameters and avoiding overfitting on few samples.

[0078] Next, the present invention proposes a forged patch generative adversarial loss for capturing the distribution of local shallow information of the image. This loss function can encourage the generator network to further optimize the repair details. After the input information is input into the forged patch discriminator E(·), E(·) outputs a two-dimensional tensor of size w'×h'. Each point of the two-dimensional tensor corresponds to the confidence of the slice within the slice range corresponding to the receptive field size (N×N) of E(·), that is, to what extent this image slice is considered to be an image slice of a real sample. Before calculating the forged patch generative adversarial loss, first construct the corresponding label map whose scale size is the same as the output of E(·). First, use the receptive field of E(·) to crop the input binary mask and the picture according to the receptive field. The cropped sub-slices R a，b and 1 ≤ a ≤ h′, 1 ≤ b ≤ m′. The construction formula for the corresponding label map is:

[0079]

[0080] If is not 0, it indicates that the corresponding sub-slice R a，b region contains the pixel points generated by the generator G(·), and it can be considered that this slice is a forged slice. Conversely, this slice is considered a real slice. So the corresponding X a，b = 1 means that the slice under the corresponding receptive field region is a forged slice, and X a，b = 0 is considered a real slice.

[0081] The calculation process of slice cropping: First, when constructing the forged patch discriminator. By statistically calculating the kernel size, stride, and padding value of all convolutional kernels in this network, the receptive field and receptive region corresponding to each unit of the final output can be obtained. This receptive region can be understood as each sub-region calculated by a filtering window during the filtering process. The window size of the filtering window is the corresponding receptive field size, and the filtering stride is the cumulative product of the strides of all convolutional kernels and downsampling operations. Similarly, the calculation of the padding value size of the filtering window is also obtained through accumulation. In this way, a corresponding filter without learning parameters can be constructed, and during the filtering process, each window is cropped.

[0082] The calculation formula for the corresponding forged patch generation adversarial loss is as follows:

[0083]

[0084] Among them, to simplify the formula, the subscript i is omitted here. The objective function for constraining the parameters of the forged patch discriminator E(·); is the objective function for constraining the parameters of the generator G(·). X is the constructed label map, and its scale size is the same as the output size of. Under the constraint of the forged patch generation adversarial loss, in order to be able to distinguish between the picture blocks containing generated pixel points and the picture blocks of real samples. The forged patch discriminator not only needs to learn the distribution of real pixel points but also needs to consider the consistency between the generated pixel points and the surrounding pixel points. This provides a more explicit and strict constraint for the discriminator. Helps the discriminator quickly capture the difference between real image blocks and generated image blocks. This also makes the generator G(·) have to consider the situation of the generated pixel points and the surrounding area, rather than simply making the generated pixel points be able to approach the distribution of real samples in terms of pixel points, thereby enhancing the visual quality of the model in repairing details.

[0085] Perceptual loss L lpips has the following formula:

[0086]

[0087] where F(·) is the feature extractor of the pre-trained classification network model. (For LPIPS loss, see: R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, "The unreasonable effectiveness of deep features as a perceptual metric," in Proc. CVPR, 2018, pp. 586 - 595.).

[0088] By jointly constraining the mapping space generative adversarial loss, the forged patch generative adversarial loss, and the perceptual loss, the formula for the final objective function of the generator is:

[0089]

[0090] θ g θ d θ e respectively represent the generator network, the mapping space discriminator network, and the forged patch discriminator network; by jointly training the generator network, the mapping space discriminator network, and the forged patch discriminator network, the formula is as follows:

[0091]

[0092]

[0093]

[0094] That is, first optimize the parameters in the mapping space discriminator network D(·) and the forged patch discriminator network E(·) respectively. Then optimize the parameters in the generator network G(·); during the inference process, the generator network G(·) is optimized through multiple iterations to achieve image inpainting based on residual learning; the mapping space discriminator network and the forged patch discriminator network are used to distinguish real and fake samples during training, enabling the model to learn the distribution of real samples through the generative adversarial approach.

[0095] According to the loss value, perform backpropagation, and use the stochastic gradient descent method to update and adjust the model parameters of the generator network G(·), the mapping space discriminator D(·), and the forged patch discriminator E(·) at the q-th time and The specific steps are as follows:

[0096] According to the final objective function, using the chain rule of differentiation, calculate the gradients of the generator network G(·), the mapping space discriminator D(·), and the forged patch discriminator E(·) with respect to their model parameters at the q-th iteration and update the corresponding parameters by stochastic gradient descent:

[0097]

[0098]

[0099]

[0100] where η represents the learning rate among the hyperparameters; and represent the gradients of the parameters of the corresponding generator network G(·), mapping space discriminator D(·), and forged patch discriminator E(·) at the q-th training iteration, respectively, obtained by the chain rule of differentiation.

[0101] Step S4. Use the test set I test to evaluate the method model of the present invention. If the restoration quality index of the model in the test images is the highest at present, save the model parameters and let and represent the optimal model parameters of the current generator network G(·), mapping space discriminator D(·), and forged patch discriminator E(·). Determine whether the training has reached the maximum number of training iterations n. If it has reached the maximum number of training iterations n, end the training phase and jump to step S5; otherwise, jump to step S3.

[0102] where using the test set I test to evaluate the method model of the present invention, the specific steps are as follows:

[0103] First, generate a fixed set of binary masks according to the test set Traverse the data. For each traversal, take a sample from the test set I test and the set of binary masks I mask at the same index. Calculate each sample using the calculation method described in step S3, but do not participate in the loss value and gradient calculations, nor perform backpropagation, and do not update the parameters. Obtain the corresponding set of restoration results For the test set I test and the set of restoration results Y testCalculate the Fr′echet inception distance metric. (FID, see: M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “GANs trained by a two-time-scale update rule converge to a local nash equilibrium,” in Proc. NIPS, 2017, pp. 6626 - 6637.). Here, define the optimal model network parameters and The FID metric of is fid best If fid best > fid q Then let fid best = fid q . The binary mask set I mask is generated only once and is used in subsequent tests all the time to ensure the same conditions for each test.

[0104] Step S5, obtain the model parameters of the generator network G(·), the mapping space discriminator D(·), and the forged patch discriminator E(·). Combine the optimal generator network model G(·) and the iterative residual learning framework proposed in this paper, and through multiple iterative forward propagations, realize the iterative residual repair of images.

[0105] As can be seen from the above technical description, implementing the embodiments of the present invention has the following beneficial effects:

[0106] 1. Compared with existing image repair methods, the present invention can train an image repair model based on a small number of samples, and this repair model can ensure the visual quality of the repaired images, having good value and prospects in practical applications;

[0107] 2. The present invention proposes a few-shot image inpainting method based on iterative residual learning. Based on the generative adversarial network architecture, this method first introduces an inpainting framework of iterative residual learning, enabling the model to only predict the residual information between the inpainted region and the content of the original image using the previous inpainting results at each stage. Under this training mechanism, the model has better generalization ability, achieving good inpainting effects not only on large-sample datasets but also excellent inpainting quality on few-shot datasets. The present invention introduces a mapping space generative adversarial loss, using the encoded features of a pre-trained classifier, enabling the mapping space discriminator to capture the global semantic relationships of the image and learn global semantic features without introducing excessive learning parameters, thus realizing the distribution learning of the overall semantic content of the image. Secondly, the present invention proposes a forged patch generative adversarial loss. Based on the receptive field of the discriminator for forged patches, the image is divided into multiple partially overlapping image patches. The forged patch discriminator network is encouraged to not only distinguish the pixel distributions between the inpainted image and the original image but also consider the consistency between the generated pixels and the surrounding pixels, providing a more explicit and strict constraint for the discriminator. This constraint also encourages the generator network to learn the pixel details, structures, and textures of the original samples, thereby enhancing the quality of the inpainted image.

[0108] To verify the inpainting effect of the technology of the present invention, the effectiveness of the invention will be verified through algorithm comparison:

[0109] This method verifies the effectiveness of the algorithm by making comparisons on the publicly available multi-sample image datasets CelebA-HQ (28,000 for training, 2,000 for testing), Paris Street View (14,900 for training, 100 for testing), as well as the few-shot image datasets CHASE_DB1 (18 for training, 10 for testing), anime face (90 for training, 30 for testing), and Animal-Face dog (309 for training, 80 for testing). The present invention is compared with 4 advanced inpainting algorithms in the recent two years, namely Recurrent Feature Reasoning (RFR), Co-mod-GAN (CMOD), Lama, and MAT.

[0110] The restoration comparison on the multi-sample image dataset is shown in Tables 1 and 2. Among them, "10-20%" means that the binary mask of the image to be restored accounts for 10%-20% of the entire image size; "Center" means a central square mask with a mask of 25%. As can be seen from Tables 1 and 2, the present invention can achieve good restoration effects in the multi-sample dataset. For example, on the Paris Street View dataset, the present invention can obtain an FID metric value of 58.08 at a mask ratio of 50-60%, which is 8.72 lower than the FID metric value of 66.80 of MAT.

[0111] Table 1 Comparison of FID Metrics on the CelebA-HQ Dataset

[0112]

[0113] Table 2 Comparison of FID Metrics on the Paris Street View Dataset

[0114]

[0115] The restoration comparison on the few-shot image dataset is shown in Tables 3, 4, and 5. As can be seen from Tables 3, 4, and 5, the present invention can achieve good restoration effects in various mask cases on the few-shot image dataset. For example, at a mask ratio of 50-60% on the anime face dataset, the present invention can obtain an FID metric value of 65.05, which is 17.77 lower than the FID metric value of 82.82 of the RFR method.

[0116] Table 3 Comparison of FID Metrics on the CHASE_DB1 Dataset

[0117]

[0118] Table 4 Comparison of FID Metrics on the anime face Dataset

[0119]

[0120] Table 5 Comparison of FID Metrics on the Animal-Face dog Dataset

[0121]

[0122] The restoration visual effect of the present invention is as Figure 3 shown. The present invention demonstrates the restoration effects on the multi-sample image dataset and the few-shot image dataset. The method of the present invention can have good restoration performance on both the multi-sample and few-shot image datasets.

[0123] Those of ordinary skill in the art can understand that all or part of the steps in the method of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium, such as ROM / RAM, disk, optical disc, etc.

[0124] The above-disclosed is only a preferred embodiment of the present invention, and of course, it cannot be used to limit the scope of the rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention still fall within the scope covered by the present invention.

Claims

1. A few-shot image inpainting method based on iterative residual learning, which constructs an image inpainting model to perform inpainting processing on an input image, characterized in that, The image inpainting model is obtained through at least the following steps: Step S1: Obtain a training set and a test set; among them, the training set is used to train and adjust the model parameters, and the test set is used to evaluate and test the model; Step S2: Construct a network model for few-shot image inpainting based on iterative residual learning; this network model at least includes a generator network, a mapping space discriminator, and a forged patch discriminator network; among them, the generator network uses the inpainting result of the previous iteration and the corresponding binary mask to further predict the residual information between the inpainted area and the original sample; a new round of inpainting result is obtained by combining the residual information; the mapping space discriminator network is used to perform generative adversarial training in the encoding space of the pre-trained classifier network, aiming to make the samples generated by the generator network have the same data distribution as the real samples in the pre-trained encoding space; the forged patch discriminator network focuses on the consistency of shallow information and assists the samples generated by the generator to have the same distribution as the real samples in terms of color, texture, and structure; initialize the models of all networks; Step S3: In any training process, first generate a binary mask with the same size as the training image, then erase the image information at the corresponding positions in the mask, and use the mask and the image as the initial input information; the model then performs multiple iterative optimizations on the input information; during the training process, each optimization is accompanied by the calculation of the loss value and the update of the model parameters; while in the test phase, the loss value is not calculated; In any optimization iteration, the inpainting result of the previous optimization and the binary mask are input into the generator network together; the generator predicts the residual information between the currently input inpainted area and the real sample according to the input information; this residual information is combined with the inpainting result of the previous time to obtain the optimized inpainting result; After obtaining the inpainting result of the current iterative optimization, the mapping space generative adversarial loss, the forged patch generative adversarial loss, and the perceptual loss calculate the loss value and perform backpropagation; the model uses the stochastic gradient descent method to update and adjust the parameters of the generator network, the mapping space discriminator, and the forged patch discriminator network; Step S4: Use the test set to evaluate the model; if the inpainting quality index of the model in the test image is the highest currently, save the model parameters; determine whether the training is over, if it is over, jump to step S5, otherwise jump to step S3; Step S5: Obtain the optimal model parameters of the generator network, the mapping space discriminator, and the forged patch discriminator network, and use the generator network as the image inpainting model for image inpainting based on iterative residual learning; In step S3, the calculation formula of the mapping space generative adversarial loss is as follows: in, is the objective function used to optimize the parameters of the mapping space discriminator D(·); is the objective function used to optimize the parameters of the generator G(·); Indicates that the input is I gt The result of seeking expectation is Similarly; ReLU(·) is a linear rectification function; P(·) is a feature extractor of a pre-trained classification model; In the step S3, after the input information is input into the forgery patch discriminator E(·), E(·) outputs a two-dimensional tensor with a size of w′×h′; each point of the two-dimensional tensor corresponds to the confidence of the slice range corresponding to the receptive field size (N×N) of E(·), that is, to what extent this image slice is considered to be an image slice of a real sample; before calculating the forgery patch generation adversarial loss, first construct the corresponding label map Its scale size is the same as the output of E(·); first, use the receptive field of E(·) to crop the input binary mask and the picture according to the receptive field; obtain the cropped sub-slice R a,b and 1≤a≤h′, 1≤b≤m′; the construction formula of the corresponding label map is: If is not 0, it indicates that the corresponding sub-slice R a,b region contains the generated pixel points, and it can be considered that this slice is a forged slice; on the contrary, this slice is considered a genuine slice; therefore, the corresponding X a,b = 1 means that the slice under the corresponding receptive field region is a forged slice, and X a,b = 0 is considered a genuine slice; The corresponding calculation formula of the forged patch generative adversarial loss is as follows: Among them, The objective function for constraining the parameters of the forgery patch discriminator E(·); is the objective function for constraining the parameters of the generator G(·); X is the constructed label map, and its scale size is the same as the output size of 2. The few-shot image inpainting method based on iterative residual learning according to claim 1, wherein, In step S1, both the training set and the test set are image sets and do not require image labels.

3. The few-shot image inpainting method based on iterative residual learning according to claim 1, wherein In the step S2, the generator G is constructed by a convolutional neural network and a Transformer; the mapping space discriminator D uses the features encoded by the pre-trained classification network model P as input and is used for binary classification determination; the forged patch discriminator network E is composed of a convolutional neural network, and the output is a two-dimensional tensor, and each point of the tensor corresponds to the distribution confidence of the image patch contained in the receptive field; the higher the scores of the outputs of the mapping space discriminator D and the forged patch discriminator network E, the higher the confidence, and the more the samples are close to the distribution of the real data.

4. The few-shot image inpainting method based on iterative residual learning according to claim 1, wherein In the step S3, the initialization formula of the input image is: I in = I gt ⊙(1 - M), Among them, I in represents the preprocessed input image; I gt represents the real sample; M is the corresponding binary mask, 1 is used to indicate the defective area, 0 represents the known area, and the known area will not be changed; ⊙ here represents element-wise multiplication; In the case of the t-th iterative optimization, the prediction formula of the corresponding residual information is: where, Δ t is the predicted residual information; is the repair result after the previous iteration optimization; when t = 1, the corresponding input repaired image is the preprocessed input image I in , that is Utilize the residual information Δ t and the previous repair result to obtain the optimized repair result, expressed as: Among them, in the inpainting area specified by the binary mask, the previous inpainting result and the residual information predicted this time will be added, and the known area reuses the original input information without change.

5. The few-shot image inpainting method based on iterative residual learning according to claim 1, characterized in that, In the step S3, by jointly constraining the mapping space generative adversarial loss, the forged patch generative adversarial loss, and the perceptual loss, the formula for the final objective function of the generator is obtained as: Among them, θ g , θ d and θ e respectively represent the parameters of the generator network, the mapping space discriminator network, and the forged patch discriminator network; by jointly training the generator network, the mapping space discriminator network, and the forged patch discriminator network, the formula is as follows: That is, first, the parameters in the mapping space discriminator network D(·) and the forged patch discriminator network E(·) are optimized separately; then, the parameters in the generator network G(·) are optimized; in the inference process, the generator network G(·) is optimized through multiple iterations to achieve image inpainting based on residual learning; the discriminator network is used to distinguish true and false samples during training, and enables the model to learn the distribution of real samples through the generative adversarial method.

Images