Image Inpainting Method Based on Multi-Granularity Dilated Convolution Neural Network

Abstract

The present invention discloses an image inpainting method based on a multi-granularity dilated convolutional neural network. First, the data set is divided into a training set and a test set, and preprocessing is performed to construct the image to be inpainted. Secondly, the image to be inpainted in the training set is input into the encoder to obtain the feature map F in . Then the feature map F in is input into the multi-granularity residual module to extract the multi-scale feature map in the image. Multiple multi-granularity residual modules are cascaded to form a multi-granularity feature extraction module to obtain the feature map F low . Finally, the feature map F low is input into two channel adaptive upsampling convolutional modules, and then input into a convolutional module to obtain the image I with the missing part filled completely, and the parameters are optimized through the image-level loss and the feature-level loss. The present invention enhances the feature extraction ability, makes the features of the inpainted image richer, and improves the accuracy of the filled pixels through the adaptive pixel rearrangement technology.

Description

Technical Field

[0001] The present invention relates to the field of image inpainting based on deep learning, and particularly to an image inpainting method based on a multi-granularity dilated convolutional neural network. Background Art

[0002] Image inpainting is a technology aimed at restoring, repairing or improving damaged or missing images, and has wide applications in the fields of digital image processing, medical imaging, security monitoring, printing media, etc. At present, based on advanced deep learning technologies, image inpainting for large-area or irregular missing regions has been greatly improved. Pathak et al. proposed a context encoding method to optimize the network model through pixel-level reconstruction loss and adversarial loss. On this basis, various improved methods have been proposed. For example, a two-stage model from rough to fine. Song et al. introduced additional auxiliary information to guide the correct repair of missing content, such as information on segmentation prediction, edge connection, and structure reconstruction. Compared with traditional methods, the above methods better handle the problem of image inpainting with large holes. However, they require training multiple networks, and the artifacts generated by the first network will propagate to the second network.

[0003] For the problem of image inpainting with a large missing area, an effective image inpainting method needs to 1) be able to extract semantic information in the image and then generate reasonable image content; 2) maintain the continuity of the edges of the inpainted area; 3) generate clear and orderly texture information. However, existing image inpainting methods cannot reasonably fill in large missing areas of images based on their deep semantic information, resulting in relatively blurred inpainted images; furthermore, existing image inpainting methods have problems of relying on multiple neural networks and complex training processes. Summary of the Invention

[0004] In view of the above technical problems, the present invention proposes an image inpainting method based on a multi-granularity dilated convolutional network. The network architecture of the present invention consists of three parts, namely an encoder, a multi-granularity feature extraction module, and a decoder. First, the encoder maps the input image to a high-dimensional feature space through convolutional layers; then, the multi-granularity feature extraction module further extracts multi-granularity high-level semantic features of the image; finally, the decoder reconstructs the complete image from the image features. The present invention designs a multi-grained residual (MGR) module, which is applied to the middle part of the network architecture. Different-scale image features are extracted through multi-granularity branches, and the multi-granularity image features are fused into more advanced features through an adaptive fusion module. The present invention designs a channel adaptive shuffling (CAS) module, which is applied to the decoder of the network architecture. By recombining the feature elements of different channels, the resolution of the feature map is increased, and the image features are decoded into the image pixel space. The technical architecture of the present invention adopts a generative network based on multi-granularity dilated convolution, and the training strategy is simple and efficient, and can fully extract multi-level information in the image, and can effectively solve the above problems.

[0005] The specific steps of the method of the present invention include:

[0006] S1. Data preprocessing: The data set is divided into a training set and a test set, the pixel values are normalized to the interval [-1, 1], and the images are scaled to a unified spatial resolution using an interpolation method. Construct the inpainting image I in , the formula is:

[0007]

[0008] where I gt is the real complete image in the training set, represents element-wise multiplication, I m is the binarized mask image, and the area with a value of 1 in I m is the known area, and the area with a value of 0 in I m is the area to be inpainted.

[0009] S2. Input the inpainting image I in in the training set into the encoder, and the encoder maps the input image I in to a high-dimensional feature space to obtain the feature map F in, the encoder is implemented by three convolutional modules, and each convolutional module consists of a convolutional layer, a batch normalization layer, and a ReLU activation function layer. Among them, the convolutional stride of the first convolutional module is set to 1, and the convolutional strides of the second and third convolutional modules are set to 2 to achieve downsampling of the feature map.

[0010] S3. Input the feature map F output in step S2 in into the multi-granularity residual module to extract the multi-scale feature map F of the image, and perform the following operations:

[0011] S31. Input the feature map F in into the fine-grained feature extraction branch to obtain the fine-grained feature map F fine :

[0012] F fine = f rn (…f ri …(f r2 (f r1 (F in ))))

[0013] where f ri represents the i-th fine-grained dilated convolution module, which includes a dilated convolution layer, a batch normalization layer, and a ReLU activation function layer; for each dilated convolution layer in this module, set its dilation rate to a relatively small value that is relatively prime, so as to achieve the extraction of multi-level fine-grained features.

[0014] S32. Input the feature map F in into the coarse-grained feature extraction branch to obtain the coarse-grained feature map F coarse :

[0015] F coarse = f cn (…f ci …(f c2 (f c1 (F in ))))

[0016] where f ci represents the i-th coarse-grained dilated convolution module, which includes a dilated convolution layer, a batch normalization layer, and a ReLU activation function layer; for each dilated convolution layer in this module, set its dilation rate to a relatively large value that is relatively prime and greater than the dilation rate set for the corresponding i-th fine-grained dilated convolution module, so as to achieve the extraction of multi-level coarse-grained features.

[0017] S33. Concatenate the fine-grained feature map and the coarse-grained feature map in the channel dimension and input them into the 1×1 convolutional layer f 1×1 (·) to obtain the convolutional feature map Fc 。

[0018] S34. Adaptively learn the channel weights of the convolutional feature map F c to obtain the weight vector A, which specifically includes the following sub-steps:

[0019] (1) Perform average pooling on the convolutional feature map F c to obtain the channel-based description D ∈ R Q×1×1 , where Q represents the number of channels, and the i-th element d i in D is obtained by averaging the i-th channel C i , and its calculation formula is:

[0020]

[0021] where H and W represent the height and width of the feature map.

[0022] (2) Perform downsampling and upsampling operations on the channel description D to filter out unimportant information, and its calculation formula is:

[0023] D′ = f up (f down (D))

[0024] where f up is the upsampling operation, and f down is the downsampling operation.

[0025] (3) Input D′ into the activation function sigmoid layer f sig to obtain the channel-related weight vector A:

[0026] A = f sig (D′)

[0027] S35. Apply the weight vector A to the convolutional feature map F c , and combine it with the residual skip connection to obtain the output F of the multi-granularity residual module:

[0028]

[0029] where, represents element-wise multiplication.

[0030] S4. Construct a multi-granularity feature extraction module by cascading multiple multi-granularity residual modules to fully extract rich multi-granularity features in the image, and obtain the low-resolution feature map F low , and each multi-granularity residual module executes steps S31 to S35.

[0031] S5. Apply the feature map F lowInput into the decoder. Specifically, it is first input into two sequentially connected channel adaptive upsampling convolution modules, and the upsampled feature map is input into a convolution module to obtain the image I with the missing part filled completely.

[0032] The channel adaptive upsampling convolution module specifically includes the following sub-steps:

[0033] S51. Input the feature map F low into the convolution layer for feature extraction to obtain the feature map F'. low .

[0034] S52. Perform pixel rearrangement (Pixel Shuffling) on the feature map F' low to obtain the high-resolution feature map F h . Through this step, the spatial resolution of the feature map is doubled, and the number of channels is reduced by 4 times.

[0035] S53. Execute step S34 on the feature map F h to obtain the channel attention vector A', and apply the channel attention to the feature map to obtain the feature map F' h :

[0036]

[0037] S54. Input the feature map F' h into the batch normalization layer and the leaky ReLU activation function layer to obtain the high-resolution feature map F output by the upsampling convolution module high .

[0038] S6. Calculate the loss, and optimize the parameters in the image restoration model of the present invention through the image-level loss and the feature-level loss. The specific steps include:

[0039] S61. The image-level loss includes the L1 reconstruction loss L hole and L valid , the multi-scale structural similarity loss L ms and the total variation loss L TV . The specific calculation formula is:

[0040]

[0041] where I is the image generated by the model, I gt is the real complete image in the training set, I com is the image composed of the generated missing area and the real existing area, I m is the mask image, where 1 represents the existing area and 0 represents the missing area, and E[·] represents taking the average. is the element-wise multiplication operator, N is the number of layers of the multi-resolution image pyramid, and W i is the weight of the i-th layer, and represents the i-th layer image pair, and are the pixel values at the positions of (x + 1, y), (x, y), and (x, y + 1) in the image I com respectively.

[0042] S62. The feature-level loss includes the high-level feature reconstruction loss L feat and the style loss L style , and its calculation formula is:

[0043]

[0044] where, represents the output of the first three pooling layers in the VGG-16 network pre-trained on the ImageNet dataset, and G(·) is the Gram matrix of the feature map.

[0045] S63. Weight different loss terms to obtain the overall loss function:

[0046] L = λ hole L hole + λ valid L valid + λ ms L ms + λ TV L TV + λ feat L feat + λ style L style

[0047] where, λ hole , λ valid , λ ms , λ TV , λ feat and λ style are the weights of the corresponding loss terms respectively.

[0048] S64. Calculate the model gradient through the loss value and update the parameters in the model.

[0049] Advantages of the present invention: The present invention maps the input image to a high-dimensional feature space through an encoder, designs different granularity feature extraction branches in the multi-granularity feature extraction module to further extract multi-granularity high-level semantic features of the image, and realizes feature map upsampling through an adaptive pixel rearrangement technique in the decoder, and reconstructs a complete image through the feature map. The present invention has the following benefits: 1. The multi-granularity feature extraction module is used to extract multi-granularity high-level semantic features of the image, enhancing the feature extraction ability of the model and making the features of the restored image richer; 2. The accuracy of the filled pixels is improved through the adaptive pixel rearrangement technique. Description of the Drawings

[0050] Figure 1 Image inpainting architecture diagram based on multi-granularity dilated convolutional neural network;

[0051] Figure 2 Structural diagram of channel adaptive upsampling module;

[0052] Figure 3 Image inpainting effect diagram of the method of the present invention for irregular missing regions;

[0053] Figure 4 Image inpainting effect diagram of the method of the present invention for regular missing regions;

[0054] Figure 5 User example effect diagram of the method of the present invention. Detailed Embodiment

[0055] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0056] As a preferred implementation form of the present invention, an image inpainting method based on a multi-granularity dilated convolutional neural network is provided, and its architecture is as Figure 1 shown, including the following steps:

[0057] S1. Data preprocessing: Divide the data set into a training set and a test set, normalize the pixel values to the interval [-1, 1], and scale the image to a unified spatial resolution of 256*256 using bicubic interpolation. Construct the image I to be inpainted in , and the formula is:

[0058]

[0059] where I gt is the real complete image in the training set, represents element-wise multiplication, I m is the binarized mask image, Im The area with a median value of 1 is the known area, I m The area with a median value of 0 is the area to be repaired.

[0060] S2. Input the image I to be repaired in the training set in into the encoder, and the encoder maps the input image I in to a high-dimensional feature space, which is implemented by three convolutional modules. Each convolutional module consists of a convolutional layer, a batch normalization layer (batchnormalization), and a ReLU activation function layer. Among them, the convolutional stride of the first convolutional module is set to 1, and the convolutional strides of the second and third convolutional modules are set to 2 to achieve downsampling of the feature map.

[0061] S3. Input the feature map F output in step S2 in into the multi-granularity residual module to extract the multi-scale feature map F of the image, and perform the following operations:

[0062] S31. Input the feature map F in into the fine-grained feature extraction branch to obtain the fine-grained feature map F fine :

[0063] F fine = f r3 (f r2 (f r1 (F in )))

[0064] where f ri represents the i-th fine-grained dilated convolution module. The fine-grained feature extraction branch described in this embodiment is composed of three cascaded dilated convolution modules, and their dilation rates are respectively set to 1, 2, and 3. It includes a dilated convolutional layer, a batch normalization layer, and a ReLU activation function layer; for each dilated convolutional layer in this module, its dilation rate is set to relatively small coprime values to achieve the extraction of multi-level fine-grained features.

[0065] S32. Input the feature map F in into the coarse-grained feature extraction branch to obtain the coarse-grained feature map F coarse :

[0066] F coarse = f c3 (f c2 (f c1 (F in )))

[0067] where f ciDenote the i-th coarse-grained dilated convolution module. In this embodiment, the coarse-grained feature extraction branch is composed of three cascaded dilated convolution modules, and their dilation rates are set to 3, 5, and 7 respectively. It includes a dilated convolution layer, a batch normalization layer, and a ReLU activation function layer; for each dilated convolution layer in this module, its dilation rate is set to a relatively large value that is relatively prime, so as to realize the extraction of multi-level coarse-grained features.

[0068] S33. Concatenate the fine-grained feature map and the coarse-grained feature map in the channel dimension and input them into the 1×1 convolutional layer f 1×1 (·) to obtain the convolutional feature map F c :

[0069] F c = f 1×1 (<F fine , F coarse >)

[0070] S34. Adaptively learn the channel weights of the convolutional feature map F c , which specifically includes the following sub-steps:

[0071] (1) Perform average pooling on the convolutional feature map F c to obtain the channel-based description D∈R Q×1×1 , where Q represents the number of channels, and the i-th element d i in D is obtained by averaging the i-th channel C i . Its calculation formula is:

[0072]

[0073] where H and W represent the height and width of the feature map.

[0074] (2) Perform downsampling and upsampling operations on the channel description D to filter out unimportant information. Its calculation formula is:

[0075] D′ = f up (f down (D))

[0076] where f up is the upsampling operation, and f down is the downsampling operation.

[0077] (3) Input D′ into the activation function sigmoid layer f sig to obtain the channel-related weight vector A:

[0078] A = f sig (D′)

[0079] S35. Apply the weight vector A to the convolutional feature map F c, and combined with the residual skip connection to obtain the output F of the multi-granularity residual module:

[0080]

[0081] Among them, represents element-wise multiplication.

[0082] S4. Eight multi-granularity residual modules are cascaded to form a multi-granularity feature extraction module, which fully extracts rich multi-granularity features in the image. Each multi-granularity residual module executes steps S31 to S35.

[0083] S5. Input the low-resolution feature map F obtained in step S4 low into the decoder. First, the low-resolution feature map F low is input into the channel adaptive upsampling convolution module, as Figure 2 shown, which specifically includes the following sub-steps:

[0084] S51. Input the feature map F low into the convolutional layer for feature extraction to obtain the feature map F' low .

[0085] S52. Perform pixel shuffling on the feature map F' low to obtain the high-resolution feature map F h . Through this step, the spatial resolution of the feature map is doubled, and the number of channels is reduced by 4 times.

[0086] S53. Execute step S34 on the feature map F h to obtain the channel attention vector A', and apply the channel attention to the feature map to obtain the feature map F' h :

[0087]

[0088] S54. Input the feature map F' h into the batch normalization layer and the leaky ReLU activation function layer to obtain the high-resolution feature map F output by the upsampling convolution module high .

[0089] S6. Input the high-resolution feature map output in step S5 into the second channel adaptive upsampling convolution module to further double its spatial resolution.

[0090] S7. Input the high-resolution feature map output in step S6 into the convolutional module to obtain the image I with the missing part filled completely.

[0091] S8. Calculate the loss, and optimize the parameters in the image inpainting model of the present invention through the image-level loss and the feature-level loss. The specific steps include:

[0092] S81. The image-level loss includes the L1 reconstruction loss L hole and L valid , the multi-scale structural similarity loss L ms and the total variation loss L TV . The specific calculation formula is:

[0093]

[0094] where I is the image generated by the model, I gt is the real complete image in the training set, I com is the image combined by the generated missing region and the real existing region, I m is the mask image, where 1 represents the existing region and 0 represents the missing region, E[·] represents taking the average, is the element-wise multiplication operator, N is the number of layers of the multi-resolution image pyramid, W i is the weight of the i-th layer, and represent the i-th layer image pair, and are the pixel values at the positions of (x + 1, y), (x, y) and (x, y + 1) in the image I com respectively.

[0095] S82. The feature-level loss includes the high-level feature reconstruction loss L feat and the style loss L style . Its calculation formula is:

[0096]

[0097] where represents the output of the first three pooling layers in the VGG-16 network pre-trained on the ImageNet dataset, and G(·) is the Gram matrix of the feature map.

[0098] S83. Weight different loss terms to obtain the overall loss function:

[0099] L = λ hole L hole + λ valid L valid + λ ms L ms + λ TV L TV + λ feat L feat + λ style Lstyle

[0100] Among them, λ hole , λ valid , λ ms , λ TV , λ feat and λ style are the weights corresponding to the respective loss terms, set as λ hole = 6, λ valid = 1, λ ms = 4, λ TV = 0.1, λ feat = 0.05, λ style = 120.

[0101] S84. Calculate the model gradient through the loss value and update the parameters in the model.

[0102] Example:

[0103] The steps of this example are the same as those in the specific implementation, and will not be elaborated here. The following shows some implementation processes and implementation results.

[0104] The technology of the present invention is implemented on the following three datasets: 1) Paris StreetView: including 14,900 training images and 100 test images; 2) CelebA-HQ: using the first 2,000 images as the test set and 28,000 images as the training set; 3) Places2: including images of various scenes, using six scene images among them, including valleys, tundras, mountain trails, mountains, canyons, and mountain peaks. Each scene includes 5,000 images as the training set and 100 images as the test set.

[0105] To verify the effectiveness of the technology of the present invention, Figure 3 , Figure 4 , Figure 5 respectively show the image restoration effect diagrams after training the technology of the present invention on different datasets. Among them, Figure 3 is the filling effect diagram for irregular missing regions. The first column is the real image, the second column is the missing image to be restored, and the third column is the image filled by the technology of the present invention; Figure 4 is the filling effect diagram for regular missing regions. The first column is the real image, the second column is the missing image to be restored, and the third column is the image filled by the technology of the present invention; Figure 5 is a user example, where the first column is the real image, the second column is the image after the user specifies the repair region, with the white region being the user-specified region, and the third column is the image filled by the technology of the present invention. The technology of the present invention can reasonably fill the missing image region with clear structural texture in the image content.

Claims

1. An image restoration method based on a multi-granularity atrous convolutional neural network, characterized in that: The following steps are involved: S1. Divide the dataset into training set and test set, perform preprocessing, and construct the image to be repaired I in ; S2. The image to be repaired in the training set I in Input to the encoder to get the feature map F in ; S3. The feature map F in Input to the multi-granularity residual module to extract the multi-scale feature map F in the image; the specific process is as follows: S31. The feature map F in Input to the fine-grained feature extraction branch to obtain the fine-grained feature map F fine : F fine =f rn (…f ri …(f r2 (f r1 (F in )))) Among them, f ri represents the i-th fine-grained dilated convolution module, which includes a dilated convolution layer, a batch normalization layer, and a ReLU activation function layer; for each dilated convolution layer in the module, its dilation rate is set to a mutually prime value; S32. The feature map F in Input to the coarse-grained feature extraction branch to obtain the coarse-grained feature map F coarse : F coarse =f cn (…f ci …(f c2 (f c1 (F in )))) Among them, f ci represents the i-th coarse-grained dilated convolution module, which includes a dilated convolution layer, a batch normalization layer, and a ReLU activation function layer; for each dilated convolution layer in the module, its dilated rate is set to a mutually prime value that is greater than the dilated rate set for the corresponding i-th fine-grained dilated convolution module; S33. The fine-grained feature map and the coarse-grained feature map are cascaded in the channel dimension and then input into the 1×1 convolution layer f 1×1 (·), and obtain the convolution feature map F c ; S34. Convolution feature map F c Adaptively learn its channel weights to obtain the weight vector A; S35. Apply the weight vector A to the convolution feature map F c , and combined with the residual jump connection to obtain the output F of the multi-granularity residual module: in, represents element-wise product; S4. By cascading multiple multi-granularity residual modules, a multi-granularity feature extraction module is formed to obtain the feature map F low ; S5. The feature map F low Input decoder: First, it passes through two sequentially connected channel adaptive upsampling convolution modules, and then inputs into a convolution module to obtain the image I with the missing parts filled in; The specific process of the channel adaptive upsampling convolution module is as follows: S51. The feature map F low Input to the convolution layer for feature extraction to obtain the feature map F′ low ; S52. Feature graph F′ low Rearrange pixels to get feature map F h , the spatial resolution of the feature map is doubled, and the number of channels is reduced by 4 times; S53. For feature map F h Execute step S34 to obtain the channel attention vector A′, apply the channel attention to the feature map, and obtain the feature map F′ h : S54. The feature map F′ h Input to the batch normalization layer and the leaky ReLU activation function layer to obtain the feature map F output by the upsampling convolution module high ; S6. Calculate the loss and optimize the parameters through image-level loss and feature-level loss.

2. The image restoration method based on multi-granularity atrous convolutional neural network according to claim 1 is characterized in that: In step S1, the preprocessing includes: normalizing the image pixel values ​​to the interval [-1, 1], and scaling the image to a uniform spatial resolution using an interpolation method; The image to be repaired I is constructed in Specifically: Among them, I gt is the real complete image in the training set, represents the element-wise product, I m is the binary mask image, I m The area with a median value of 1 is a known area. m The area with a median value of 0 is the area to be repaired.

3. The image restoration method based on multi-granularity atrous convolutional neural network according to claim 2 is characterized in that: In step S2, the encoder is implemented by three convolution modules, each of which consists of a convolution layer, a batch normalization layer and a ReLU activation function layer. The convolution step size of the first convolution module is set to 1, and the convolution step sizes of the second and third convolution modules are set to 2 to achieve downsampling of the feature map.

4. The image restoration method based on multi-granularity atrous convolutional neural network according to claim 3 is characterized in that: The specific process of step S34 is as follows: (1) Convolutional feature map F c Perform average pooling to obtain a channel-based description D∈R Q×1×1 , Q represents the number of channels, the i-th element d in D i For the i-th channel C i The calculation formula is: Among them, H and W represent the height and width of the feature map; (2) Downsampling and upsampling operations are performed on the channel description D, and the calculation formula is: D′=f up (f down (D)) Among them, f up is the upsampling operation, f down It is a downsampling operation; (3) Input D′ into the activation function sigmoid layer to obtain the channel-related weight vector A.

5. The image restoration method based on multi-granularity atrous convolutional neural network according to claim 4 is characterized in that: The specific process of step S6 is as follows: S61. The image level loss includes L1 reconstruction loss L hole and L valid , multi-scale structural similarity loss L ms and the total variation loss L TV , the calculation formula is: Among them, I is the image generated by the model, I com is an image composed of the generated missing area and the real existing area, E[·] represents the average, is an element-by-element multiplication operator, N is the number of multi-resolution image pyramid layers, and W i The weight of the i-th layer, and represents the i-th layer image pair, and I com The pixel values ​​at the positions (x+1, y), (x, y) and (x, y+1) are in the middle coordinates; S62. The feature level loss includes feature reconstruction loss L feat and style loss L style , the calculation formula is: in, represents the output of the first three pooling layers in the VGG-16 network pre-trained on the ImageNet dataset, and G(·) is the Gram matrix of the feature map; S63. Weight the different loss terms to get the overall loss function: L=λ hole L hole +λ valid L valid +λ ms L ms +λ TV L TV +λ feat L feat +λ style L style Among them, λ hole , valid , ms , TV , feat and λ style are the weights of the corresponding loss items respectively; S64. Calculate the model gradient through the loss value and update the parameters in the model.

Images