An improved image inpainting method of generative adversarial network

Abstract

The application provides a method for image restoration based on a generative adversarial network, which is based on a U-Net and an attention mechanism, uses an improved U-Net to extract large-range and multi-scale feature information, uses an attention mechanism network to obtain global feature information, and through verification, the restoration algorithm has better restoration effect and has wide application value.

Description

Technical Field

[0001] This invention belongs to the field of computer image processing technology, and specifically relates to an image inpainting method based on generative adversarial networks. Background Technology

[0002] Image inpainting is a technique that predicts a complete image from a missing image. It is a highly competitive and advanced technology with applications across various industries. With the development of computer image processing technology, the methods for image inpainting are becoming increasingly diverse. Digital image inpainting methods are mainly divided into two directions: traditional inpainting algorithms and deep learning-based inpainting methods. Traditional inpainting methods primarily fall into two categories: diffusion-based and sample-based methods. Bertalmio et al. (Proceedings of the 27th annual conference on Computer graphics and interactive techniques) simulated manual image inpainting and proposed a diffusion-based image inpainting method for the first time, achieving certain inpainting results. Diffusion-based methods spread background information to the missing region, but the algorithm has a high computational cost and requires a significant amount of runtime. To address this issue, Chan et al. (Mathematical models for local nontexture in painting) proposed the Total Variational (TV) model, which significantly improved the inpainting speed, but the continuity of the inpainting results was insufficient. He Kai et al. added a threshold parameter to the TV algorithm, which accelerated the model training speed and reduced the runtime. Sample-based methods search the background region to obtain sample blocks with high similarity to the missing region, which are then used to fill in the missing area. For example, Barnes et al. (Arandomized correspondence algorithm for structural image editing) used a fast nearest neighbor approach with a sample matching algorithm, reducing model complexity and improving computational speed. Criminisi et al. (Toyama K. Region filling and object removal by exemplar-based image inpainting) used confidence and embedded boundary information to determine the priority of the filling order, searched for the pixel block with the highest similarity to the complete region block in the image to be filled, updated the boundary and confidence of the block to be filled, and completed the image completion. Traditional image inpainting methods have achieved certain results in restoring ancient paintings with small missing areas, but the restoration effect is poor when there are large missing areas.

[0003] In recent years, deep learning, represented by Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), has been widely used in fields such as natural language processing, computer vision, medicine, and transportation. Many experts and scholars have adopted basic modules of deep learning and applied them to research on data and image completion. Wang et al. (New inpainting algorithm based on simplified context encoders and multi-scale adversarial network) proposed an image inpainting model based on an autoencoder structure combined with multi-scale convolution, using a Generative Adversarial Network (GAN) as its model framework. This improved the model's local feature extraction ability and achieved certain restoration results. However, due to the model's lack of ability to extract global feature information, some restoration results showed disordered contours. Edger et al. (Au-net based discriminator for generative adversarial networks) proposed a GAN model based on an improved discriminator using a U-Net. This model improved the global consistency of the generated data through feedback from the discriminator's iteration results. However, the model did not consider generator optimization, resulting in poor model learning ability. Summary of the Invention

[0004] The present invention mainly overcomes the shortcomings of the prior art, and the purpose of the present invention is to provide an image inpainting method based on generative adversarial networks.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0006] Step 1: Acquire the original image and preprocess it. The specific processing method is as follows:

[0007] (1) Preprocess the original image, uniformly set it to n×n size and perform cropping, rotation and mirror flipping operations to expand the data;

[0008] (2) Create binary images of defects of different sizes and shapes, and synthesize them with the original images to form defect images;

[0009] (3) Divide the preprocessed image dataset into a training set and a test set, and use the preprocessed images as the training target;

[0010] Step 2: Construct dilated convolution gated residual blocks. The entire dilated convolution gated residual block uses 3×3 convolutions and ReLU activation function. The computational structure of the module is as follows:

[0011] x1=r(Dcon2(x in ))

[0012] x2 = r(Dcon4(x1))

[0013] x3 = r(Dcon8(x2))

[0014] x4=r(Dcon 16 (x3))

[0015] x out =g(x1)·x1+g(x2)·x2+g(x3)·x3+g(x4)·x4

[0016] In the formula, x in and x out These represent the input and output images of the model, respectively. r() indicates a ReLU linear transformation, Dcon() represents a 3×3 dilated convolution operation with its index indicating the dilation rate, and g(x) represents the weight coefficients obtained after gating the input data. The gating process uses the Sigmoid function for calculation, and the formula for calculating the Sigmoid function is as follows:

[0017]

[0018] Step 3: The overall image inpainting model is based on a generative adversarial network framework and uses a two-stage approach to inpaint the image. The specific processing method is as follows:

[0019] (1) The first-stage network of the generator generates a coarse inpainted image. The overall structure is a U-Net network. Its encoding part contains n dilated convolution gated residual blocks proposed in step 2. n is the number of layers in the encoding part of the U-Net network; n is a positive integer.

[0020] (2) The second-stage network of the generator uses a spatial attention mechanism to generate the final repaired image. The defect area image generated in the first stage is used as the foreground and the known area image is used as the background. The cosine similarity between the foreground and the background is calculated pixel by pixel. The background with a high cosine similarity is selected to generate the foreground. The formula for calculating the cosine similarity is as follows:

[0021]

[0022] In the formula, (x1, y1) represents the pixels of the background image, and (x2, y2) represents the pixels of the foreground image. Indicates the cosine similarity between the background and foreground;

[0023] (3) Use the weighted loss of minimum absolute error loss and bulldozer distance loss as the loss function, and regularize it using a penalty function;

[0024] The formula for calculating the minimum absolute error loss is:

[0025]

[0026] In the formula, Represents the minimum absolute error loss, y i and f(x) i () represent the real image and the estimated image, respectively;

[0027]

[0028] In the formula, L wgan Represents the distance loss of the bulldozer. Let represent the set of all joint distributions of (x, y);

[0029] The formula for calculating the joint loss function is:

[0030]

[0031] Where L is the overall weighted loss function, For l1 loss, L wgan For the distance loss of the bulldozer, and Here, λ represents the weighting coefficient, and λ represents the penalty coefficient of the penalty function. Denotes the L2 norm of ω;

[0032] (4) Use the training set data obtained in step 1 to input into the network model constructed in step 4 and this step, use the loss function for training optimization, and stop model training when the loss function no longer decreases, and save the training parameters.

[0033] Step 4: Peak signal-to-noise ratio (PSNR) is an evaluation metric that reflects the degree of distortion between the generated image and the real image. Its unit is dB, and its calculation formula is:

[0034]

[0035] Where MSE is the root mean square error, MAX I It is the maximum value of the image pixels, and PSNR is the peak signal-to-noise ratio of the image.

[0036] Structural similarity measures the structural similarity between two images. A higher value indicates better image quality. Structural similarity uses the mean to estimate brightness, the standard deviation to estimate contrast, and the covariance to estimate structural similarity. The calculation formula is as follows:

[0037]

[0038]

[0039]

[0040] SSIM(A,B)=L(A,B)×C(A,B)×S(A,B)

[0041] In the formula, μ A and μ B σ is the average pixel value of images A and B. A and σ B σ is the standard deviation of pixels in images A and B. AB C1, C2, and C3 are constants, and SSIM is the image structural similarity.

[0042] The test set data obtained in step 1 is input into the trained network model to obtain the repaired image. Finally, the repair effect is evaluated using objective evaluation metrics.

[0043] Beneficial effects:

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] Based on U-Net and the attention mechanism, this invention improves U-Net to extract large-scale, multi-scale feature information and uses an attention mechanism network to obtain global feature information. As a result, the repair algorithm of this invention has a better repair effect and has a wide range of application value. Attached Figure Description

[0046] Figure 1 This is a flowchart of an image inpainting process based on generative adversarial networks.

[0047] Figure 2 Drilled convolution gated residual blocks;

[0048] Figure 3 Improved U-Net architecture diagram;

[0049] Figure 4 Attention mechanism network structure diagram; Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings.

[0051] An image inpainting method based on generative adversarial networks includes the following computational steps:

[0052] Step 1: Acquire the original image and preprocess it. The specific processing method is as follows:

[0053] (4) Preprocess the original image, uniformly set it to 256×256 size and perform cropping, rotation and mirror flipping operations to expand the data;

[0054] (5) Generate binary images of defects of different sizes and shapes, and synthesize them with the original image to form a defect image;

[0055] (6) Divide the preprocessed image dataset into a training set and a test set, and use the preprocessed images as the training target;

[0056] Step 2: Construct dilated convolution gated residual blocks. The entire dilated convolution gated residual block uses 3×3 convolutions and ReLU activation function. The computational structure of the module is as follows:

[0057] x1=r(Dcon2(x in ))

[0058] x2 = r(Dcon4(x1))

[0059] x3 = r(Dcon8(x2))

[0060] x4=r(Dcon 16 (x3))

[0061] x out =g(x1)·x1+g(x2)·x2+g(x3)·x3+g(x4)·x4

[0062] In the formula, x in and x out These represent the input and output images of the model, respectively. r() indicates a ReLU linear transformation, Dcon() represents a 3×3 dilated convolution operation with its index indicating the dilation rate, and g(x) represents the weight coefficients obtained after gating the input data. The gating process uses the Sigmoid function for calculation, and the formula for calculating the Sigmoid function is as follows:

[0063]

[0064] Step 3: The overall image inpainting model is based on a generative adversarial network framework and uses a two-stage approach to inpaint the image. The specific processing method is as follows:

[0065] (1) The first-stage network of the generator generates a coarse inpainted image. The overall structure is a U-Net network, and its encoding part contains four dilated convolution gated residual blocks proposed in step 2.

[0066] (2) The second-stage network of the generator uses a spatial attention mechanism to generate the final repaired image. The defect area image generated in the first stage is used as the foreground and the known area image is used as the background. The cosine similarity between the foreground and the background is calculated pixel by pixel. The background with a high cosine similarity is selected to generate the foreground. The formula for calculating the cosine similarity is as follows:

[0067]

[0068] In the formula, (x1, y1) represents the pixels of the background image, and (x2, y2) represents the pixels of the foreground image. Indicates the cosine similarity between the background and foreground;

[0069] (3) Use the weighted loss of minimum absolute error loss and bulldozer distance loss as the loss function, and regularize it using a penalty function;

[0070] The formula for calculating the minimum absolute error loss is:

[0071]

[0072] In the formula, Represents the minimum absolute error loss, y i and f(x) i () represent the real image and the estimated image, respectively;

[0073]

[0074] In the formula, L wgan Represents the distance loss of the bulldozer. Let represent the set of all joint distributions of (x, y);

[0075] The formula for calculating the joint loss function is:

[0076]

[0077] Where L is the overall weighted loss function, For l1 loss, L wgan For the distance loss of the bulldozer, and Here, λ represents the weighting coefficient, and λ represents the penalty coefficient of the penalty function. Denotes the L2 norm of ω;

[0078] (4) Use the training set data obtained in step 1 to input into the network model constructed in step 4 and this step, use the loss function for training optimization, and stop model training when the loss function no longer decreases, and save the training parameters.

[0079] Step 4: Peak signal-to-noise ratio (PSNR) is an evaluation metric that reflects the degree of distortion between the generated image and the real image. Its unit is dB, and its calculation formula is:

[0080]

[0081] Where MSE is the root mean square error, MAX IIt is the maximum value of the image pixels, and PSNR is the peak signal-to-noise ratio of the image.

[0082] Structural similarity measures the structural similarity between two images. A higher value indicates better image quality. Structural similarity uses the mean to estimate brightness, the standard deviation to estimate contrast, and the covariance to estimate structural similarity. The calculation formula is as follows:

[0083]

[0084]

[0085]

[0086] SSIM(A,B)=L(A,B)×C(A,B)×S(A,B)

[0087] In the formula, μ A and μ B σ is the average pixel value of images A and B. A and σ B σ is the standard deviation of pixels in images A and B. AB C1, C2, and C3 are constants, and SSIM is the image structural similarity.

[0088] The test set data obtained in step 1 is input into the trained network model to obtain the repaired image. Finally, the repair effect is evaluated using objective evaluation metrics.

[0089] Step 5: The implementation effects of this invention are as follows. The effects of the implementation method of this invention are shown in Table 1 below, which presents the optimization effect of this invention on defect repair.

[0090] Table 1. Repair results of different algorithms at different overall missing rates.

[0091] Tab.1Evaluation index values ​​of repair results of different algorithms under different overall center missing rates

[0092]

[0093] As can be seen from the table above, this method achieves better repair results than the algorithm used in comparison for defects of different sizes.

[0094] This invention provides an image inpainting method based on generative adversarial networks (GANs). It utilizes U-Net and an attention mechanism, employing an improved U-Net to extract large-scale, multi-scale feature information and an attention mechanism network to acquire global feature information. The inpainting algorithm of this invention achieves superior inpainting results. This invention can more completely repair defective images and has broad application value.

[0095] The above description is not intended to limit the present invention in any way. Although the present invention has been disclosed through the above embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. An improved image inpainting method using generative adversarial networks, Its features include the following steps: Step 1: Acquire the original image and preprocess it. The specific processing method is as follows: Preprocess the original images and set them uniformly. Resize and perform cropping, rotation, and mirroring operations to expand the data; Binary images of defects of different sizes and shapes are artificially generated and synthesized with the original image to form a defect image; The preprocessed image dataset is divided into a training set and a test set, with the preprocessed images used as the training target. Step 2: Construct dilated convolution gated residual blocks. The entire dilated convolution gated residual block uses... The convolution uses the ReLU activation function, and the module's computational structure is as follows: ； ； ； ； ； In the formula, and These represent the input and output images of the model, respectively. This indicates a ReLU linear transformation. represent The dilated convolution operation is defined by its index as the dilation rate. This represents the weight coefficients obtained after performing a gating operation on the input data. The gating process uses the Sigmoid function for calculation, and the formula for the Sigmoid function is as follows: ； Step 3: The overall image inpainting model is based on a generative adversarial network framework and uses a two-stage approach to inpaint the image. The specific processing method is as follows: (1) The first-stage network of the generator generates a coarsely repaired image. The overall structure is a U-Net network, and its encoding part includes The dilated convolution gated residual block proposed in step 2 n is the number of layers in the coding portion of the U-Net network; n is a positive integer. (2) The second-stage network of the generator uses a spatial attention mechanism to generate the final repaired image. The defect area image generated in the first stage is used as the foreground and the known area image is used as the background. The cosine similarity between the foreground and the background is calculated pixel by pixel. The background with a large cosine similarity is selected to generate the foreground. The formula for calculating the cosine similarity is as follows: ； In the formula ( , ) represents the pixels of the background image, ( , ) represents the pixels of the foreground image. Indicates the cosine similarity between the background and foreground; (3) Use the weighted loss of minimum absolute error loss and bulldozer distance loss as the loss function, and regularize it using a penalty function; The formula for calculating the minimum absolute error loss is: ； In the formula, Represents the minimum absolute error loss. and These represent the real image and the estimated image, respectively. ； In the formula, Represents the distance loss of the bulldozer. Indicates all The set of joint distributions; (4) Use the training set data obtained in step 1 to input into the network model constructed in step 4 and this step, use the loss function for training optimization, and stop model training when the loss function no longer decreases and save the training parameters. Step 4: Peak signal-to-noise ratio (PSNR) is an evaluation metric that reflects the degree of distortion between the generated image and the real image. Its unit is dB, and its calculation formula is: ； in, It is the root mean square error. It is the maximum value of the image pixels. It is the peak signal-to-noise ratio of the image; Structural similarity measures the structural similarity between two images. A higher value indicates better image quality. Structural similarity uses the mean to estimate brightness, the standard deviation to estimate contrast, and the covariance to estimate structural similarity. The calculation formula is as follows: ； ； ； ； In the formula, and It is the average pixel value of images A and B. and It is the standard deviation of pixels in images A and B. It is the covariance of images A and B. , , It is a constant. Image structural similarity; The test set data obtained in step 1 is input into the trained network model to obtain the repaired image. Finally, the repair effect is evaluated using objective evaluation metrics.

Images