Image inpainting method based on down-sampling confidence network

Abstract

The application discloses an image inpainting method based on a downsampling confidence network, comprising the following steps: 1) preparing an open source data set and dividing the data set into a training set, a validation set and a test set; 2) designing a downsampling confidence network, specifically taking DeepFillv2 as an image generation module, and proposing a downsampling confidence module, and adding the confidence pixels generated by the downsampling confidence module to the input image of the image generation module; 3) training and verifying the downsampling confidence network by using the training set and the validation set; 4) testing the trained downsampling confidence network by using the test set, and outputting the repaired image. The confidence pixels generated based on the downsampling confidence network obtain the information of a far distance of the image, which helps to strengthen the constraint of the center of the missing area of the output image, achieves a semantic clear result, and not only ensures excellent performance, but also better controls the calculation cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image inpainting deep learning, and in particular to an image inpainting method based on a down-sampling confidence network. BACKGROUND

[0002] In practical applications, images are often corroded by noise caused by lens dust, or scratches on old photos, or human painting, etc. Therefore, effective image inpainting can prevent the image from further deterioration, such as removing scratches on old photos. It can also be used for image editing to remove unwanted image content, such as removing raindrops, etc.

[0003] With the increasing demand for image inpainting, three types of methods have been developed, which are: 1. Traditional image inpainting algorithms that use data near the missing area of the image to repair the missing area by diffusion or known patches. However, due to the inability to understand the deep semantic of the image, there are problems of structural distortion and blurred inpainting image, which cannot meet the current inpainting requirements; 2. Deep learning-based methods, most of the existing advanced deep neural network methods use one-time padding, which can achieve good results when the missing area is small, but it is difficult to achieve ideal inpainting results for large area damaged images; 3. Progressive inpainting is proposed to solve the problem of large missing area. This solves the problem of insufficient hole center constraints. However, these methods still have the problem of structural distortion and image distortion, because the hole center not only has a strong correlation with the local area pixels, but also has a certain correlation with the distant information. Therefore, these methods either limit information to the local area, resulting in poor performance, or couple modules with networks, making it difficult to control the computational cost. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide an image inpainting method based on a down-sampling confidence network. Based on the confidence pixels generated by the down-sampling confidence module, the distant information of the image is obtained, which helps to strengthen the constraints of the center of the missing area of the output image, and achieves a semantic clear result. This not only ensures superior performance, but also better controls the computational cost.

[0005] To achieve the above purpose, the technical scheme provided by the present application is: an image inpainting method based on a down-sampling confidence network, comprising the following steps:

[0006] 1) Prepare an open source data set, and divide the data set into a training set, a validation set and a test set;

[0007] 2) design a down-sampling confidence network, which is improved based on DeepFillv2, specifically, DeepFillv2 is used as an image generation module, and a down-sampling confidence module is proposed, the confidence pixels generated by the down-sampling confidence module are added to the input image of the image generation module, the constraint of the center of the image missing area of the image generation module is strengthened, the far information in the confidence pixels is obtained, and the image achieves a semantic clear result;

[0008] 3) using the training set to train the down-sampling confidence network, using the validation set to judge the fitting effect of the network, and selecting the best network model as the finally trained down-sampling confidence network;

[0009] 4) using the test set to test the trained down-sampling confidence network, and outputting the repaired image, that is, the final prediction image.

[0010] Further, in step 1), the data set comes from Places2 data set, Paris StreetView data set and a mask data set, because the down-sampling confidence network has a consistency requirement for the picture, so it is necessary to scale the data set by the same proportion, and process it into a picture with a size of 256x256.

[0011] Further, in step 2), the down-sampling confidence network is divided into two parts: an image generation module and a down-sampling confidence module; the image generation module uses Deep Fillv2, because it uses a gated convolution, which can help identify the effectiveness of the confidence pixels generated by the down-sampling confidence module, and support different resolutions in inference; the structure of the down-sampling confidence module is similar to that of Deep Fillv2, and adopts a model structure from a coarse network to a fine network, both the coarse network and the fine network use the structure of an encoder and a decoder, the coarse network has an encoder and a decoder; the fine network has an encoder and two decoders, one decoder is used to generate a complete image, and the other decoder is used to generate a confidence matrix, in order to help the decoder generating the confidence to understand the details of image generation, a skip connection is used between the two decoders; in the fine network stage, a spectral normalized Markov discriminator is used for adversarial training, the input of which is composed of an image and a mask, and the output is a three-dimensional feature with a shape of HxWxC, H represents length, W represents width, and C represents channel number.

[0012] Further, in step 3), the overall parameters of the down-sampling confidence network are trained using an RGB three-channel image, and the steps are as follows:

[0013] First, the images in the training set are down-sampled and input into the coarse network of the down-sampling confidence module, and the specific operation is: the original input image I in , also known as a high-resolution image, is represented as:

[0014] I in =I gt ·m

[0015] In the formula, I gt This refers to real images from a public dataset; m is a binary mask image, where 0 represents a known region of the image and 1 represents an unknown region; I in The low-resolution image I′ is obtained by downsampling m. in and mask m′;

[0016] Then, I′ in Together with m′, we obtain a coarse prediction image I′ as input to a coarse network. p G(·) represents the result generated by the generator, expressed as:

[0017] I' p =G(I′) in )

[0018] Using L1 reconstruction loss to explicitly train a coarse network, its loss function L coarse Defined as:

[0019]

[0020] In the formula, L coarse It is the L1 distance between the downsampled real image and the image generated by the coarse network; I′ p (i,j) represents the pixel value in the i-th row and j-th column of the image generated by the coarse network; gt ′(i,j) represents the pixel value in the i-th row and j-th column of the downsampled real image; n and k represent the number of pixels in each row and each column, respectively;

[0021] Next, the results are passed to a finer network to generate a refined prediction image I′. r The confidence matrix c is expressed as:

[0022] I' r =G(I′) p )

[0023] c = tanh(G(I′) p ))

[0024] The downsampling confidence module uses the tanh function for prediction, with an output range of [-1, 1]. Confidence values ​​greater than 0 in the confidence matrix are set to 1 to indicate good repair quality, and confidence values ​​less than 0 are set to 0 to indicate poor repair quality. The confidence loss L... c The definition is as follows:

[0025]

[0026] In the formula, Used to represent I′ in , I′ gt The expected value of the distribution, I′ gt L represents the downsampled real image; L(·) represents the L1 loss; ξ is a hyperparameter controlling the size of the confidence region; the first term in the equation encourages the confidence matrix to be optimized in the loss L(I′). r It has a high response in the case of small values ​​because it outputs I′ from the generator. r • In option c, select the low-loss region and replace the true value I′ with the high-loss region. gt ·(1-c), L C It is expected to be smaller; the second term penalizes the all-zero confidence matrix by encouraging confident regions to cover as many missing regions as possible;

[0027] The loss function of a downsampled confidence network includes discriminator loss and generator loss, with the discriminator loss function L... D for:

[0028]

[0029] In the formula, Indicates I′ gt In P data Expected spatial distribution; Indicates I′ in In P data The expected value of the spatial distribution; ReLU represents the ReLU activation function; D(·) represents the discriminator output;

[0030] The objective function L of the generator G for:

[0031]

[0032] In the formula, Used to represent I′ in ,I′ gt The expected value of the distribution; ||I′ r -I′ gt ||1 represents I′ r and I′ gt The L1 norm of the distance between them.

[0033] Further, in step 4), the images in the test set are input into the trained down-sampling confidence network, a low-resolution image and a confidence matrix are generated by the down-sampling confidence module, then the low-resolution image and the confidence matrix generated by the down-sampling confidence module are multiplied to obtain a confidence pixel, and the confidence pixel is up-sampled and filled into the input image of the image generation module, the filled input image is input into the image generation module, the missing area of the image generation module can obtain reasonable confidence information as prior information by the down-sampling confidence module, and the network parameters are updated by using the gradient descent method in cooperation with the chain rule through the loss function, and finally the image result, that is, the repaired image I, is obtained r .

[0034] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0035] 1. The present application first proposes a down-sampling confidence network, which uses down-sampling to not only deepen the constraint of the hole center but also strengthen the correlation between the faraway pixel and the hole center pixel, thereby significantly improving the network performance.

[0036] 2. The present application improves DeepFillv2 and improves the effectiveness of the confidence pixel.

[0037] 3. The present application has a wide application space in computer vision tasks, is simple to operate, has strong adaptability, and has a broad application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The present application uses an example schematic diagram. DETAILED DESCRIPTION

[0039] The present application will be further described in detail below in combination with embodiments and drawings, but the embodiments of the present application are not limited thereto.

[0040] As shown in the figure, the present embodiment provides an image inpainting method based on a down-sampling confidence network, including the following steps: Figure 1 1) Prepare an open source data set, and divide the data set into a training set, a validation set and a test set;

[0041] The data set comes from the Places2 data set, the Paris StreetView data set and a mask data set, since the down-sampling confidence network has a consistency requirement for the picture, it is necessary to perform equal proportion scaling processing on the data set, and uniformly process the picture to 256x256.

[0042]

[0043] ​2) Design a downsampling confidence network, which is improved based on DeepFillv2, specifically, DeepFillv2 is used as an image generation module, and a downsampling confidence module is proposed, the confidence pixels generated by the downsampling confidence module are added to the input image of the image generation module, the constraint of the center of the image missing area of the image generation module is strengthened, the far information in the confidence pixels is obtained, and the image achieves a semantic clear result;

[0044] Specifically, the downsampling confidence network is divided into two parts: an image generation module and a downsampling confidence module; the image generation module uses Deep Fillv2, because it uses a gating convolution, which can help identify the effectiveness of the confidence pixels generated by the downsampling confidence module, and support different resolutions in inference; the structure of the downsampling confidence module is similar to that of Deep Fillv2, and adopts a model structure from a coarse network to a fine network, both the coarse network and the fine network use the structure of an encoder and a decoder, the coarse network has an encoder and a decoder; the fine network has an encoder and two decoders, one decoder is used to generate a complete image, and the other decoder is used to generate a confidence matrix, in order to help the decoder generating the confidence to understand the details of image generation, a skip connection is used between the two decoders; in the fine network stage, a spectral normalized Markov discriminator is used for adversarial training, the input of which is composed of an image and a mask, and the output is a three-dimensional feature with a shape of HxWxC, H represents length, W represents width, and C represents the number of channels.

[0045] 3) Use the training set to train the downsampling confidence network, use the validation set to judge the fitting effect of the network, and select the best network model as the final trained downsampling confidence network;

[0046] Specifically, the RGB three-channel image is used to train the overall parameters of the downsampling confidence network, and the steps are as follows:

[0047] First, the images in the training set are down-sampled and input into the coarse network of the downsampling confidence module, the specific operation is: the original input image I in , also known as a high-resolution image, is represented as:

[0048] I in =I gt ·m

[0049] In the formula, I gt is a real image of a public data set; m is a binary mask image with a value of 1, where 0 represents the known area of the image and 1 represents the unknown area; I in and m are down-sampled to obtain a low-resolution image I′ in and a mask m′;

[0050] Then, I′ in and m′ together as the input of the coarse network to get the coarse prediction image I′ p , G(·) represents the result generated by the generator, which is expressed as:

[0051] I′ p = G(I′ in )

[0052] The coarse network is trained explicitly using L1 reconstruction loss, and its loss function L coarse is defined as:

[0053]

[0054] In the formula, L coarse is the L1 distance between the down-sampled real image and the coarse network generated image; I' p (i,j) represents the pixel value of the i-th row and j-th column in the coarse network generated image; I' gt '(i,j) represents the pixel value of the i-th row and j-th column in the down-sampled real image; n and k represent the number of pixels in each row and each column, respectively;

[0055] Then, the result is passed to the fine network to generate the fine prediction image I′ r and the confidence matrix c, which is expressed as:

[0056] I′ r = G(I' p )

[0057] c = tanh(G(I' p ))

[0058] The down-sampling confidence module uses the tanh function as the prediction, and its output interval is [-1, 1]; the confidence values greater than 0 in the confidence matrix are set to 1, indicating good repair quality, and the confidence values less than 0 are set to 0, indicating poor repair quality; the confidence loss L c is defined as follows:

[0059]

[0060] In the formula, is used to represent the expectation of I′ in ,I′ gt distribution, I' gt represents the down-sampled real image; L(·) represents the L1 loss; ξ is a hyperparameter that controls the size of the confidence region; the first term in the formula encourages the confidence matrix to have a high response when the loss L(I′ r ) is small, because by selecting the low-loss region from the generator output I′ r ·c and replacing the high-loss region with the real value I', the loss of the whole network can be reduced.gt • (1-c), L C is expected to be smaller; the second term penalizes all-zero confidence matrices by encouraging confident regions to cover as much of the missing region as possible;

[0061] The loss function of the down-sampling confidence network includes the discriminator loss and the generator loss, and the loss function L D is:

[0062]

[0063] wherein, denotes the expectation of I' gt In P data space distribution; denotes the expectation of I' in In P data space distribution; ReLU denotes the ReLU activation function; D(·) denotes the discriminator output;

[0064] The objective function L G of the generator is:

[0065]

[0066] wherein, denotes the expectation of I' in ,I' gt distribution; ||I' r -I' gt ||1 denotes the L1 norm of the distance between I' r and I' gt .

[0067] 4) input the images in the test set into the trained down-sampling confidence network, generate a low-resolution image and a confidence matrix through the down-sampling confidence module, then multiply the low-resolution image and the confidence matrix generated by the down-sampling confidence module to obtain a confidence pixel, up-sample the filled input image to the input image of the image generation module, input the filled input image to the image generation module, the missing region of the image generation module can generate reasonable confidence information as prior information through the down-sampling confidence module, and the network parameters are updated through the gradient descent method and the chain rule through the loss function, to obtain the final image result, that is, the repaired image I r .

[0068] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited by the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application are equivalent replacement methods, and are all included in the protection scope of the present application.

Claims

1. An image inpainting method based on down-sampling confidence network, characterized in that, Comprise the following steps: 1) Prepare the open source dataset, and divide the dataset into training set, validation set and test set; 2) Design a downsampling confidence network, which is improved based on DeepFillv2, specifically DeepFillv2 is used as an image generation module, and a downsampling confidence module is proposed, the confidence pixels generated by the downsampling confidence module are added to the input image of the image generation module, the constraint of the image generation module on the center of the image missing area is strengthened, the far information in the confidence pixels is obtained, and the image reaches the result of clear semantics; The downsampling confidence network is divided into two parts: an image generation module and a downsampling confidence module; The image generation module uses Deep Fillv2 because it uses a gating convolution, which can help identify the effectiveness of the confidence pixels generated by the downsampling confidence module, and support different resolutions in inference; The structure of the downsampling confidence module is similar to that of DeepFillv2, which adopts a model structure from a coarse network to a fine network, both the coarse network and the fine network use the structure of an encoder and a decoder; The fine network has an encoder and two decoders, one decoder is used to generate a complete image, and the other decoder is used to generate a confidence matrix, in order to help the confidence decoder understand the details of image generation, a skip connection is used between the two decoders; In the fine network stage, a spectral normalization Markov discriminator is used for adversarial training, its input is composed of an image and a mask, and the output is a three-dimensional feature with a shape of HxWxC, H represents length, W represents width, and C represents channel number; 3) Use the training set to train the downsampling confidence network, use the validation set to judge the fitting effect of the network, and select the best network model as the final trained downsampling confidence network; 4) Use the test set to test the trained downsampling confidence network, output the repaired image, that is, the final prediction image; The image in the test set is input into the trained downsampling confidence network, a low-resolution image and a confidence matrix are generated through the downsampling confidence module, then the low-resolution image and the confidence matrix generated by the downsampling confidence module are multiplied to obtain a confidence pixel, the filled input image is input into the image generation module, the missing area of the image generation module can generate reasonable confidence information as prior information through the downsampling confidence module, and the network parameters are updated through the loss function using the gradient descent method in cooperation with the chain rule, and finally the image result, that is, the repaired image I, is obtained. r .

2. The method of claim 1, wherein the confidence network is a down-sampling confidence network. In step 1), the dataset comes from Places2 dataset, Paris StreetView dataset and a mask dataset, because the downsampling confidence network has a consistency requirement for the picture, it is necessary to scale the dataset by the same proportion, and process it into 256x256 pictures.

3. The method of claim 2, wherein, In step 3), the RGB three-channel image is used to train the overall parameters of the downsampling confidence network, and the steps are as follows: First, the images in the training set are down-sampled and input to the coarse network of the down-sampling confidence module. The specific operation is as follows: the original input image I in Also known as a high-resolution image, is represented as: I in = I gt ·m In the formula, I gt denotes a real image of the public dataset; m is a binary mask image with value 0 indicating known regions of the image and 1 indicating unknown regions; I in and m down-sampled to a low resolution image I' in and mask m' Then, I' in and m' together as inputs to the coarse network yields a coarse prediction image I' p G(·) denotes the result of the generator, expressed as: I' p = G(I' in ) The coarse network is trained using the L1 reconstruction loss explicitly, whose loss function L coarse is defined as: wherein L coarse is the L1 distance between the down-sampled real image and the coarse network generated image; I p (i,j) represents the pixel value of the i-th row and j-th column in the coarse network generated image; I gt '(i,j) represents the pixel value of the i-th row and j-th column in the down-sampled real image; n and k represent the number of pixels in each row and each column, respectively. The results are then passed to the fine network, generating a fine prediction image I' r and a confidence matrix c, expressed as: I' r = G(I' p ) c = tanh(G(I p )) The downsampling confidence module uses a tanh function as the prediction, and the output interval thereof is [-1, 1]; confidence values greater than 0 in the confidence matrix are set to 1 to indicate good repair quality, and confidence values less than 0 are set to 0 to indicate poor repair quality; the confidence loss L c is defined as follows: where is used to denote I in , gt the expectation of the distribution, I gt denotes the downsampled real image; L(·) denotes the L1 loss; and ξ is a hyperparameter that controls the size of the confidence region; the first term in the equation encourages the confidence matrix to have high response in cases where the loss L(I r ') is small, because by selecting low-loss regions from the generator output I r '·c and replacing high-loss regions with the real value I gt ·(1 - c), L C is expected to be smaller; the second term penalizes all-zero confidence matrices by encouraging confident regions to cover as much of the missing region as possible; The loss function of the downsampling belief network includes a discriminator loss and a generator loss, the loss function L D is: wherein represents I' gt In P data the expectation of the spatial distribution; represents I' in In P data the expectation of the spatial distribution; ReLU represents a ReLU activation function; D(·) represents a discriminator output; Objective function L of the generator G is: wherein is used to denote I in , I gt is the expectation of the distribution of I r - I gt is used to denote I r and I gt is the L1 norm of the distance between I

Images