An image inpainting method based on a multi-scale large convolutional neural network

Abstract

The application discloses an image repairing method based on a multi-scale large convolutional neural network, which can be used for repairing the texture and semantics of a damaged image, so that the damaged image can be repaired and the effect of conforming to human sensory perception can be achieved. The method comprises the following steps: according to original image data and an image mask, pre-processing is performed to obtain a to-be-repaired image which is shielded by the mask, and then a training data set is constructed; an image repairing MSLK-GAN generative adversarial network model based on a multi-scale large convolutional neural network is constructed; the constructed training data set is input into a generator of the MSLK-GAN network, and an RGB image after repairing is output; the repaired RGB image is input into a discriminator of the MSLK-GAN network to obtain the probability that each part of the image is discriminated as an original image, and the generator and the discriminator are jointly trained through a weighted loss function; and the training effect of the model MSLK-GAN is evaluated on a test set. Compared with current mainstream models, the application achieves better recovery effect in three indexes of L1 error, peak signal-to-noise ratio and structural similarity.

Description

Technical Field

[0001] This invention relates to an image inpainting method based on a multi-scale large convolutional neural network, applicable to the field of image inpainting technology in computer vision. Background Technology

[0002] Image restoration technology, primarily referring to the process of repairing and reconstructing missing content in a low-quality image based on known background information, has a history spanning several decades. In the past, image restoration technology was mainly used to restore damaged artworks, playing a crucial role in the preservation of ancient cultural relics. However, due to technological limitations, this image restoration work relied heavily on expert experience, was ineffective for large-scale damage, and suffered from irreversible damage; different experts also had varying restoration styles. Currently, the development of computer vision has facilitated the emergence of digital image restoration. This method does not alter the original image data or artifacts, offers greater flexibility, and is relatively safer. Furthermore, the application of digital image restoration is not limited to artifact restoration; when photographs are damaged or contaminated with dye, digital image restoration technology can be used to restore them, bringing practical significance to people's daily lives and work.

[0003] Goodfellow et al. proposed Generative Adversarial Networks (GANs), which divide the network into a generator and a discriminator. The data does not require labels, making it one of the most promising methods of unsupervised learning and providing a new option for image inpainting tasks. Zheng et al. proposed a diversified inpainting method. Previous studies mostly aimed to obtain a single filling of the image to be repaired, but in reality, there are multiple reasonable possibilities. Therefore, to compensate for the lack of diversity, Zheng et al. proposed a probabilistic learning framework with two parallel paths: a reconstruction path and a generation path. The reconstruction path only uses the corresponding labels to obtain prior information about the missing region and completes the reconstruction. The generation path couples the conditional prior to the distribution obtained by the reconstruction path. The two paths are trained adversarially. Furthermore, an attention mechanism that models the relationship between long and short regions is proposed to improve image consistency. Liu et al. also addressed the issue of inpainting diversity and proposed the PD-GAN (Probabilistic Diverse GAN) model. The PD-GAN model uses the Vanilla GAN framework, which assumes that the greater the diversity is closer to the center of the hole, the lower the diversity is closer to the edge of the hole because it needs to maintain connectivity with the context. This is used as a priori information to ensure the diversity of GAN, thus ensuring that the generated images are diverse and not abrupt. Summary of the Invention

[0004] Purpose of the invention: To address the existing difficulties in image restoration technology, this invention proposes an image restoration method based on a multi-scale large convolutional neural network, which improves the quality of restored images.

[0005] Technical solution: An image inpainting method based on multi-scale large convolution, including the following steps:

[0006] Step S1: Construct an RGB image dataset for image inpainting, and randomly select RGB images I from the dataset. real And the mask M in the mask dataset, generate the RGB image I to be repaired. masked ;

[0007] Step S2: Construct the generator and discriminator of the image inpainting generative adversarial network MSLK-GAN;

[0008] Step S3: Transfer the I obtained in step S1 masked The M data is used to perform a forward pass operation on the generator of the MSLK-GAN network to obtain the generator's output I. G Using I G The final restored RGB image I was obtained with M. comp The loss function of the generator is calculated, and the Adam optimizer is used to optimize and update the generator of the MSLK-GAN network. This process is iterated to obtain the optimized generator of the MSLK-GAN network.

[0009] Step S4: Convert the RGB image I obtained in step S3 into a single image. comp The input is fed into the discriminator of the MSLK-GAN network for forward computation to obtain the discriminator's decision I. comp Each part represents the probability of the original image. The loss function of the discriminator is calculated, and the Adam optimizer is used to optimize, update, and iterate the discriminator of the MSLK-GAN network to obtain the optimized discriminator of the MSLK-GAN network.

[0010] Step S5: After training the generator and discriminator of the MSLK-GAN network through steps S3 and S4, input the test set data into the generator of the MSLK-GAN network according to steps S1 and S3 to obtain the repaired RGB image.

[0011] Furthermore, step S1 specifically includes:

[0012] Step S101: Randomly select a mask M from the mask dataset and reduce its size to 256×256. Similarly, randomly select a real image I from the RGB image dataset. real The size was reduced to 256×256, and data augmentation operations such as random rotation and random cropping / scaling were performed.

[0013] Step S102: The RGB domain image I to be repaired is calculated from the mask and the real image obtained in step S101 using equation (1). masked .

[0014] I masked = (1-M)×I real +M (1)

[0015] Furthermore, step S3 specifically includes:

[0016] Step S301: Use the image mask M obtained in step S101 and the RGB domain image to be repaired obtained in step S102 as input to train the MSLK-GAN network generator, and extract multi-scale information through the MSLK module.

[0017] Step S302: Unsupervised training of the MSLK-GAN network generator is performed using L1 loss function, Style loss function, Perceptual loss function, and Generator Adversarial Loss function. During training, training stops after the loss function values ​​converge, thus obtaining the optimized and usable MSLK-GAN Generative Adversarial Network generator, and obtaining the generator output I. G ;

[0018] Step S303: The generator output I obtained in step S302 G The final repaired RGB domain image I can be obtained from the image mask M obtained in step S101 by equation (2). comp .

[0019] I comp = (1-M)×I real +M×I G (2)

[0020] Furthermore, step S4 specifically includes:

[0021] Step S401: The final repaired RGB domain image I obtained in step S303 comp It is used as input to the discriminator of the MSLK-GAN generative adversarial network for training;

[0022] Step S402: The discriminator of the MSLK-GAN network is trained unsupervised using the discriminator adversarial loss function. During the training process, the training stops after the loss function value converges, thus obtaining the optimized and usable MSLK-GAN generative adversarial network discriminator.

[0023] Furthermore, step S5 specifically includes:

[0024] Step S501: Fix the parameters of the MSLK-GAN network generator obtained by training in step S302;

[0025] Step S502: Using the RGB domain image of the test set and the image mask as input, obtain the final repaired RGB domain image I of the test set using equation (2). comp .

[0026] The beneficial effects of this invention are: it can complete high-quality restoration of damaged RGB domain images. Firstly, it uses the original RGB domain image I... real The RGB domain image I to be repaired is obtained by using the image mask M. masked Then I masked Using M as input to the MSLK-GAN network generator, the repaired RGB domain image I is obtained through computation. comp Compared with the comparison algorithm, the method proposed in this invention achieves better results in terms of L1 error, peak signal-to-noise ratio, and structural similarity of the repaired image, as tested. This indicates that the method of this invention can more effectively and with higher quality complete image restoration work. From a qualitative perspective, the restored image provided by the method of this invention is more consistent with human cognition, indicating that it can better recover the semantic information of the image to be restored. Furthermore, the large convolutional technology and multi-scale technology used provide feasible solutions for further exploration by future researchers and have further reference value. Attached Figure Description

[0027] Figure 1 This is a flowchart of the image inpainting method based on a multi-scale large convolutional neural network in Example 1.

[0028] Figure 2 This is a flowchart of the training and prediction process of the MSLK-GAN generative adversarial network in Example 1.

[0029] Figure 3 This is a structural diagram of the MSLK-GAN generative adversarial network model in Example 1.

[0030] Figure 4 This is a structural diagram of the MSLK module in Example 1.

[0031] Figure 5 This is a diagram showing the effect of repairing the RGB domain image in Example 1. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Example 1

[0034] See Figures 1-5 This invention can perform high-quality restoration of damaged RGB domain images. First, it uses the original RGB domain image I... real The RGB domain image I to be repaired is obtained by using the image mask M. masked Then I masked Using M as input to the MSLK-GAN network generator, the repaired RGB domain image I is obtained through computation. comp The repaired RGB domain image is used as input to the MSLK-GAN network discriminator, which outputs the probability that each part is a real image.

[0035] Specifically, see Figure 1 This method specifically includes:

[0036] Step S1: Construct an RGB image dataset for image inpainting, randomly selecting RGB images I from the source dataset. real The mask M in the mask dataset is preprocessed to generate the RGB image I to be repaired. masked ;

[0037] More specifically, step S1 includes: randomly selecting a mask M from the image mask dataset and reducing its size to 256×256. Similarly, randomly selecting a real image I from the RGB image source dataset... real The size was reduced to 256×256, and data augmentation operations such as random rotation, random cropping, and scaling were performed; the mask M and the real image I... real The RGB domain image I to be repaired is calculated. masked .

[0038] In this example, the image mask M is a single-channel grayscale image. A value of 1 (white) represents a portion of the image that is corrupted and needs to be restored, while a value of 0 (black) represents a portion of the image that remains consistent with the original image. To ensure the robustness of the training results, this example uses random rotation, random cropping, and scaling for data augmentation. The mask is randomly selected to guarantee the randomness of image corruption.

[0039] Step S2: Construct the generator and discriminator of the image inpainting generative adversarial network MSLK-GAN;

[0040] Step S3: Transfer the I obtained in step S1 masked The M data is used to perform a forward pass operation on the generator of the MSLK-GAN network to obtain the generator's output I. G Using I G The final restored RGB image I was obtained with M. compThe loss function of the generator is calculated, and the Adam optimizer is used to optimize and update the generator of the MSLK-GAN network. This process is iterated to obtain the optimized generator of the MSLK-GAN network.

[0041] More specifically, in this embodiment, step S3 includes: using the image mask M obtained in step S1 and the RGB domain image to be repaired as inputs to the MSLK-GAN network generator for training; performing unsupervised training of the MSLK-GAN network generator using the L1 loss function, Style loss function, Perceptual loss function, and Generator Adversarial Loss function; stopping training after the loss function values ​​converge during training, thereby obtaining an optimized and usable MSLK-GAN Generative Adversarial Network generator, and obtaining the generator output I. G The generator outputs I. G The final restored RGB domain image I can be obtained by calculating with the image mask M. comp .

[0042] Step S4: Convert the RGB image I obtained in step S3 into a single image. comp The input is fed into the discriminator of the MSLK-GAN network for forward computation to obtain the discriminator's decision I. comp Each part represents the probability of the original image. The loss function of the discriminator is calculated, and the Adam optimizer is used to optimize and update the discriminator of the MSLK-GAN network. This process is iterated to obtain the optimized discriminator of the MSLK-GAN network.

[0043] More specifically, in this embodiment, the final repaired RGB domain image I obtained in step S3 is... comp The discriminant of the MSLK-GAN generative adversarial network is trained by using the discriminant adversarial loss function. The discriminant is trained unsupervised by using the discriminant adversarial loss function. During the training process, the training stops after the loss function value converges, thus obtaining the optimized and usable MSLK-GAN generative adversarial network discriminant.

[0044] It should be noted that in this embodiment, the training of the MSLK-GAN network generator in step S3 and the training of the MSLK-GAN network discriminator in step S4 are carried out alternately, that is, the two are competing with each other and moving closer to the equilibrium point where the generator produces new samples with the same distribution as the real data.

[0045] Figure 3 The overall architecture of the MSLK-GAN model is shown. MSLK-GAN connects eight MSLK modules serially in the generator. The input data pairs include the RGB domain image to be repaired. maskedThe input data first passes through the encoder module of the generator to extract shallow features, then through the MSLK module to extract multi-scale information of the image, which is then input into the decoder of the generator. Finally, the output I of the generator is used to obtain the image mask M. G The repaired image is calculated using the image mask M and used as input to the discriminator.

[0046] Figure 4 The internal structure of the MSLK module is shown. Each MSLK module takes a 256-channel tensor as input, which is divided into four groups of 64 channels each. Different sized convolutional kernels are applied to each group to extract multi-scale information. The outputs of each group are concatenated along the channel dimension to restore the 256-channel module. Then, feature fusion is performed using a 2D convolutional layer with a 3-kernel configuration and a gated residual structure to obtain the module output.

[0047] Figure 5 The image inpainting results of MSLK-GAN are shown. The image mask dataset is divided into six parts according to the occlusion range: 0%-10%, 10%-20%, 20%-30%, 30%-40%, 40%-50%, and 50%-60%. Qualitatively, the MSLK-GAN generative adversarial network achieves better results than the control network in restoring both image texture and semantics. This is especially evident when the image to be repaired has large areas of damage, resulting in better texture continuity and semantic plausibility. As shown in Table 1, quantitatively, MSLK-GAN achieves better results in all three selected metrics: L1 error, PSNR, and SSIM. This is consistent with the results obtained from qualitative observations.

[0048] Table 1 shows the L1 error, peak signal-to-noise ratio, and structural similarity values ​​in Example 1.

[0049]

[0050] Any aspects of this invention not described in detail are well-known to those skilled in the art.

[0051] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An image inpainting method based on a multi-scale large convolutional neural network, characterized in that, The method includes the following steps: Step S1: Construct an image dataset for image inpainting, randomly selecting images from the dataset. Image masks in the image mask dataset Image preprocessing is performed to generate the image to be repaired. ; Step S2: Construct the generator and discriminator of the image inpainting generative adversarial network MSLK-GAN; Step S3: Obtain the information from step S1 and The data is fed into the generator of the MSLK-GAN network and used for forward propagation to obtain the generator's output. ,use and Obtain the final restored image ; Calculate the loss function of the generator, and use the Adam optimizer to optimize and update the generator of the MSLK-GAN network, and iterate to obtain the optimized generator of the MSLK-GAN network. Step S4: Transfer the image obtained in step S3 The input is fed into the discriminator of the MSLK-GAN network for forward computation to obtain the discriminator's decision. The probability that each part is the original image; The loss function of the discriminator is calculated, and the Adam optimizer is used to optimize and update the discriminator of the MSLK-GAN network. The process is iterated to obtain the optimized discriminator of the MSLK-GAN network. Step S5: After training the generator and discriminator of the MSLK-GAN network through steps S3 and S4, input the test set data into the generator of the MSLK-GAN network according to steps S1 and S3 to obtain the repaired RGB domain image. Specifically, step S1 includes: Step S101: Randomly select an image mask from the image mask dataset. Reduce the size to 256 256; Randomly select real images from the RGB domain image dataset. Reduce the size to 256 256, and perform data enhancement operations such as random rotation and random cropping / scaling; Step S102: The RGB domain image to be repaired is calculated from the mask and the real image obtained in step S101 using equation (1). , （1）； Step S3 specifically includes: Step S301: Apply the image mask obtained in step S101 The RGB domain image to be repaired obtained in step S102 is used as input to the MSLK-GAN network generator for training, and multi-scale information is extracted by the MSLK module. Step S302: Using Unsupervised training is performed on the MSLK-GAN network generator using loss functions, style loss function, perceptual loss function, and generator adversarial loss function. Training stops once the loss function values ​​converge, resulting in an optimized and usable MSLK-GAN generative adversarial network generator, from which the generator output is obtained. ; Step S303: Generator output obtained in step S302 and the image mask obtained in step S101 The final repaired RGB domain image can be obtained from equation (2). , （2）。 2. The image inpainting method based on a multi-scale large convolutional neural network according to claim 1, characterized in that, Step S4 specifically includes: Step S401: Obtain the final repaired RGB domain image obtained in step S303. It is used as input to the discriminator of the MSLK-GAN generative adversarial network for training; Step S402: The discriminator of the MSLK-GAN network is trained unsupervised using the discriminator adversarial loss function. During the training process, the training stops after the loss function value converges, thus obtaining the optimized and usable MSLK-GAN generative adversarial network discriminator. Step S403: Discriminator outputs judgment Each part is a probability matrix of the original image. .

3. The image inpainting method based on a multi-scale large convolutional neural network according to claim 1, characterized in that, Step S5 specifically includes: Step S501: Fix the parameters of the MSLK-GAN network generator obtained by training in step S302; Step S502: Using the RGB domain image of the test set and the image mask as input, obtain the final repaired RGB domain image of the test set using equation (2). .

Images